Sep 10 11:14:41.817433 ip-10-0-141-40 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 11:14:42.244200 ip-10-0-141-40 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:14:42.244200 ip-10-0-141-40 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 11:14:42.244200 ip-10-0-141-40 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:14:42.244200 ip-10-0-141-40 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 11:14:42.244200 ip-10-0-141-40 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:14:42.247029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.246938    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 11:14:42.250190 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250173    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.250190 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250189    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250193    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250196    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250199    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250202    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250206    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250209    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250213    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250216    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250219    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250224    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250228    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250231    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250234    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250237    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250240    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250243    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250246    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250249    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250252    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.250259 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250255    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250257    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250260    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250263    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250265    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250269    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250272    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250276    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250278    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250281    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250283    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250286    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250288    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250291    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250294    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250297    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250299    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250302    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250304    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.250741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250307    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250309    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250313    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250316    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250319    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250321    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250323    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250326    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250328    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250331    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250334    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250337    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250340    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250343    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250347    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250350    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250353    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250355    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250358    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.251221 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250360    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250363    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250366    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250368    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250371    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250374    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250376    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250378    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250381    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250384    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250387    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250390    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250392    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250394    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250397    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250399    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250402    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250404    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250407    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250409    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.251680 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250412    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250414    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250417    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250419    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250422    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250424    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250426    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250429    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250432    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250435    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250438    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250440    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250443    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250445    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250448    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250450    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250453    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250455    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250457    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.252175 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.250460    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252311    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252319    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252323    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252326    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252329    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252331    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252334    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252337    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252340    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252342    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252345    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252347    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252350    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252352    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252355    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252358    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252360    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252363    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252366    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.252662 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252368    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252371    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252373    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252376    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252379    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252381    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252383    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252386    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252388    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252390    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252393    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252395    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252399    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252402    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252407    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252410    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252414    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252417    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252419    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.253176 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252422    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252425    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252427    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252430    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252432    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252434    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252437    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252440    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252442    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252444    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252447    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252449    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252451    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252454    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252456    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252459    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252461    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252464    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252466    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.253651 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252469    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252472    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252475    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252478    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252480    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252483    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252486    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252491    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252495    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252499    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252502    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252505    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252508    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252511    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252513    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252515    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252519    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252521    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252524    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252526    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.254125 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252529    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252531    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252534    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252536    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252539    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252541    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252543    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252546    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252548    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252551    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252553    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252556    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252558    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252561    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252563    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252566    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252568    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252571    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252573    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252576    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.254629 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252579    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.252582    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253785    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253793    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253796    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253799    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253802    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253806    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253809    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253812    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253814    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253817    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253820    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253822    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253825    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253830    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253833    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253835    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253838    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.255130 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253840    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253843    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253845    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253848    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253850    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253853    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253856    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253858    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253860    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253863    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253865    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253868    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253871    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253874    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253877    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253880    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253884    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253886    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253888    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.255618 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253891    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253893    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253896    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253898    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253900    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253903    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253905    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253908    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253910    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253912    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253915    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253917    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253920    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253922    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253924    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253927    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253929    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253932    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253935    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.256075 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253939    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253942    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253945    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253948    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253951    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253953    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253957    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253960    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253963    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253965    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253968    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253972    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253975    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253978    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253981    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253984    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253987    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253989    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253992    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253994    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.256539 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253997    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.253999    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254001    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254004    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254006    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254010    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254013    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254016    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254018    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254021    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254023    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254026    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254028    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254030    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254033    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254035    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254038    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254040    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254043    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.257124 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254045    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254048    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254050    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.254052    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254140    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254187    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254200    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254204    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254209    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254212    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254217    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254221    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254224    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254227    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254230    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254233    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254236    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254240    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254242    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254245    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254248    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.254251    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255095    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255099    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 11:14:42.257641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255103    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255106    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255109    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255113    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255116    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255120    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255123    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255127    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255130    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255133    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255136    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255139    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255144    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255147    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255156    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255159    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255162    2578 flags.go:64] FLAG: --enable-server="true"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255165    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255172    2578 flags.go:64] FLAG: --event-burst="100"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255175    2578 flags.go:64] FLAG: --event-qps="50"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255178    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255181    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255184    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255187    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 11:14:42.258239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255190    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255193    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255199    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255202    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255205    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255208    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255211    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255214    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255217    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255219    2578 flags.go:64] FLAG: --feature-gates=""
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255223    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255226    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255229    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255232    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255235    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255238    2578 flags.go:64] FLAG: --help="false"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255241    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255244    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255247    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255250    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255253    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255256    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255259    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 11:14:42.258809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255268    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255270    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255274    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255276    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255279    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255282    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255285    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255288    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255291    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255294    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255296    2578 flags.go:64] FLAG: --lock-file=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255299    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255302    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255305    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255311    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255314    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255317    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255319    2578 flags.go:64] FLAG: --logging-format="text"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255322    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255326    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255329    2578 flags.go:64] FLAG: --manifest-url=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255332    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255341    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255344    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255348    2578 flags.go:64] FLAG: --max-pods="110"
Sep 10 11:14:42.259422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255351    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255355    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255357    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255360    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255363    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255366    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255369    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255376    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255385    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255388    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255391    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255394    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255400    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255402    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255405    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255408    2578 flags.go:64] FLAG: --port="10250"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255411    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255414    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0793995c9d08a5ca5"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255417    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255420    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255423    2578 flags.go:64] FLAG: --register-node="true"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255426    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255430    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255433    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 10 11:14:42.260048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255436    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255439    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255443    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255446    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255449    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255452    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255455    2578 flags.go:64] FLAG: --runonce="false"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255458    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255461    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255464    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255467    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255470    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255473    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255476    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255479    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255482    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255485    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255494    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255497    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255500    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255503    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255506    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255511    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255514    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255516    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 11:14:42.260674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255525    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255530    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255533    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255536    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255538    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255542    2578 flags.go:64] FLAG: --v="2"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255546    2578 flags.go:64] FLAG: --version="false"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255550    2578 flags.go:64] FLAG: --vmodule=""
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255554    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.255557    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255679    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255683    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255686    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255689    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255692    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255695    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255698    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255700    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255703    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255706    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255708    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255711    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.261342 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255714    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255718    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255720    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255723    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255726    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255728    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255731    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255734    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255736    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255739    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255743    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255747    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255750    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255753    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255756    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255759    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255762    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255765    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255768    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.261890 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255770    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255773    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255776    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255778    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255781    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255783    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255785    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255788    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255790    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255793    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255796    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255798    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255800    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255803    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255807    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255809    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255812    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255814    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255817    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255819    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.262371 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255822    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255824    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255828    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255831    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255834    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255836    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255839    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255841    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255844    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255846    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255849    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255851    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255853    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255856    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255858    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255861    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255864    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255866    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255868    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.262855 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255871    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255873    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255876    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255878    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255881    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255883    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255886    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255889    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255892    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255894    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255897    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255900    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255902    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255904    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255907    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255909    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255914    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255917    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255919    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.263484 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255922    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255924    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255927    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255929    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255931    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255934    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255936    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255938    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.255941    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.263987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.256895    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:14:42.265848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.265818    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 11:14:42.265997 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.265986    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 11:14:42.266851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.266817    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 11:14:42.266941 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.266858    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 11:14:42.267119 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267107    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.267119 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267118    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267122    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267125    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267129    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267132    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267135    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267138    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267141    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267143    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267146    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267149    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267151    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267154    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267157    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267159    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267162    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267165    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267167    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267170    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267172    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.267180 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267175    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267178    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267182    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267186    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267189    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267192    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267195    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267197    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267201    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267204    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267206    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267210    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267215    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267218    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267221    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267223    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267226    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267228    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267231    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267233    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.267709 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267236    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267238    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267241    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267243    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267246    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267249    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267251    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267254    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267256    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267259    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267262    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267264    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267266    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267269    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267272    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267275    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267278    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267281    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267285    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267288    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.268231 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267293    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267296    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267299    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267302    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267305    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267307    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267310    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267314    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267317    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267320    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267323    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267326    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267329    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267332    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267335    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267337    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267340    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267343    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267346    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.268727 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267348    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267351    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267354    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267356    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267359    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267361    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267363    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267367    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267370    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267372    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267375    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267377    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267380    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267382    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267385    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267387    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267390    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267392    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.269205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267395    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.267400    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267520    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267525    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267528    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267531    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267534    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267536    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267539    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267542    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267544    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267546    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267549    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267551    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267555    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:14:42.281972 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267557    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267560    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267562    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267565    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267567    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267570    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267573    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267575    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267578    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267580    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267584    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267586    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267589    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267591    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267594    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267596    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267599    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267602    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267607    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:14:42.282395 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267611    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267614    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267617    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267620    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267623    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267625    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267629    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267631    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267634    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267637    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267640    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267642    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267644    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267647    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267650    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267652    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267655    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267657    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267660    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267663    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:14:42.282866 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267665    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267668    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267670    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267673    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267675    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267678    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267680    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267683    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267685    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267687    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267690    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267692    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267695    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267697    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267700    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267703    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267706    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267709    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267712    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:14:42.283386 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267715    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267718    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267720    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267723    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267725    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267728    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267730    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267733    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267735    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267738    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267741    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267743    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267746    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267748    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267751    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267753    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267756    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267758    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267761    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267763    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:14:42.283845 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267765    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267768    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267770    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267773    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267775    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267777    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267780    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:42.267782    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.267787    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.267902    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.270861    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.272245    2578 server.go:1019] "Starting client certificate rotation"
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.272343    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 11:14:42.284464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.272382    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 11:14:42.284787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.275529    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 11:14:42.284787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.276708    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 11:14:42.289728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.289711    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 10 11:14:42.293179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.293162    2578 log.go:25] "Validated CRI v1 image API"
Sep 10 11:14:42.294029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.294015    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 11:14:42.298267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.298245    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7deedee1-016c-451d-9b5d-5277612ed316:/dev/nvme0n1p3 bd3680c5-6670-446e-8f99-868c98f981dc:/dev/nvme0n1p4]
Sep 10 11:14:42.298349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.298265    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 11:14:42.303652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.303533    2578 manager.go:216] Machine: {Timestamp:2026-09-10 11:14:42.30223932 +0000 UTC m=+0.377564975 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096268 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e880dc7f83632f8a8ed8b9dbcb0e SystemUUID:ec26e880-dc7f-8363-2f8a-8ed8b9dbcb0e BootID:2907f7a8-e19c-4908-8dab-abe8e48c118c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cf:91:40:ce:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cf:91:40:ce:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:c6:5c:2b:5f:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 11:14:42.303652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.303649    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 11:14:42.303791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.303737    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 11:14:42.304994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.304966    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 11:14:42.305183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.304997    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 11:14:42.305908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.305896    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 11:14:42.305943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.305911    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 11:14:42.305943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.305920    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 11:14:42.306784 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.306768    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 11:14:42.308664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.308652    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 11:14:42.311018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.311002    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 11:14:42.313215 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313202    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 11:14:42.313336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313326    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 11:14:42.313369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313356    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 11:14:42.313396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313368    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 11:14:42.313396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313382    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 11:14:42.313396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313393    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 10 11:14:42.313486 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.313402    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 11:14:42.314681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.314668    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 11:14:42.314738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.314690    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 11:14:42.319141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.319118    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 11:14:42.322501 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.322487    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 11:14:42.322600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.322517    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 11:14:42.324178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324165    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324187    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324194    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324199    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324208    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324214    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 11:14:42.324229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324228    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 11:14:42.324417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324233    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 11:14:42.324417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324241    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 11:14:42.324417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324247    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 11:14:42.324417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324256    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 11:14:42.324417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.324261    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 11:14:42.325181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.325169    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 11:14:42.325233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.325184    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 11:14:42.326145 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.326117    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 11:14:42.326230 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.326119    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 11:14:42.328178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.328159    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 11:14:42.329293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.329279    2578 server.go:1268] "Started kubelet"
Sep 10 11:14:42.329378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.329353    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 11:14:42.329435 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.329363    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 11:14:42.329435 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.329429    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 11:14:42.330114 ip-10-0-141-40 systemd[1]: Started Kubernetes Kubelet.
Sep 10 11:14:42.331246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.331167    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 11:14:42.331631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.331617    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 11:14:42.334503 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.334481    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 11:14:42.335525 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.334374    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-40.ec2.internal.18d3f1627e98ca31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-40.ec2.internal,UID:ip-10-0-141-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-40.ec2.internal,},FirstTimestamp:2026-09-10 11:14:42.329250353 +0000 UTC m=+0.404576008,LastTimestamp:2026-09-10 11:14:42.329250353 +0000 UTC m=+0.404576008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-40.ec2.internal,}"
Sep 10 11:14:42.336876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.336864    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 11:14:42.336921 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.336883    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 11:14:42.336979 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.336966    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.337825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.337798    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 11:14:42.337825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.337818    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 11:14:42.337968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.337830    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 11:14:42.337968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.337945    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 11:14:42.337968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.337959    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 11:14:42.338109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338002    2578 factory.go:55] Registering systemd factory
Sep 10 11:14:42.338109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338016    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 10 11:14:42.338256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338234    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-znzlp"
Sep 10 11:14:42.338357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338299    2578 factory.go:153] Registering CRI-O factory
Sep 10 11:14:42.338357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338313    2578 factory.go:223] Registration of the crio container factory successfully
Sep 10 11:14:42.338357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338357    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 11:14:42.338503 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338375    2578 factory.go:103] Registering Raw factory
Sep 10 11:14:42.338503 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338388    2578 manager.go:1196] Started watching for new ooms in manager
Sep 10 11:14:42.338974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.338958    2578 manager.go:349] Starting recovery of all containers
Sep 10 11:14:42.348006 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.347978    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 11:14:42.348242 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.348197    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 11:14:42.348632 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.348607    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-znzlp"
Sep 10 11:14:42.349645 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.349624    2578 manager.go:354] Recovery completed
Sep 10 11:14:42.355532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.355514    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.357545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.357526    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.357624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.357566    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.357624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.357579    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.358179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358165    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 11:14:42.358235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358180    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 11:14:42.358235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358187    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.358235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358198    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 11:14:42.358881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358864    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.358956 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358898    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.358956 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.358911    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.360274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.360263    2578 policy_none.go:49] "None policy: Start"
Sep 10 11:14:42.360319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.360280    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 11:14:42.360319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.360290    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 11:14:42.360428 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.360329    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-40.ec2.internal.18d3f16280488501  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-40.ec2.internal,UID:ip-10-0-141-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-40.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-40.ec2.internal,},FirstTimestamp:2026-09-10 11:14:42.357544193 +0000 UTC m=+0.432869847,LastTimestamp:2026-09-10 11:14:42.357544193 +0000 UTC m=+0.432869847,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-40.ec2.internal,}"
Sep 10 11:14:42.362317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.362305    2578 policy_none.go:47] "Start"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.403710    2578 manager.go:337] "Starting Device Plugin manager"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.403749    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.403763    2578 server.go:87] "Starting device plugin registration server"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.404032    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.404044    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.404151    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.404223    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.404231    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.404819    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 11:14:42.422317 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.404862    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.447758 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.447717    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 11:14:42.449042 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.449022    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 11:14:42.449196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.449100    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 11:14:42.449196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.449141    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 11:14:42.449196 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.449192    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 11:14:42.452052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.452025    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:42.505032 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.504943    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.505997 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.505978    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.506107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.506019    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.506107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.506032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.506107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.506058    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.519425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.519404    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.519496 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.519430    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-40.ec2.internal\": node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.545553 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.545524    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.549443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.549427    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"]
Sep 10 11:14:42.549532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.549504    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.550646 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.550631    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.550724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.550662    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.550724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.550678    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.552999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.552984    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.553140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.553182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553143    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.553889 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553874    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.553955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553895    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.553955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553905    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.553955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553924    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.553955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553944    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.554125 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.553957    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.556227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.556213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.556287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.556239    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:14:42.557050 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.557030    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:14:42.557154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.557063    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:14:42.557154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.557077    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:14:42.578903 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.578879    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-40.ec2.internal\" not found" node="ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.583437 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.583420    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-40.ec2.internal\" not found" node="ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.640459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.640430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.640459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.640461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.640585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.640482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d6d72dcc0110b2d026681ad98a3f85b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-40.ec2.internal\" (UID: \"3d6d72dcc0110b2d026681ad98a3f85b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.646546 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.646524    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.741659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.741759 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.741759 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d6d72dcc0110b2d026681ad98a3f85b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-40.ec2.internal\" (UID: \"3d6d72dcc0110b2d026681ad98a3f85b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.741759 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.741759 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d6d72dcc0110b2d026681ad98a3f85b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-40.ec2.internal\" (UID: \"3d6d72dcc0110b2d026681ad98a3f85b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.741759 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.741746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f14ea0a754358c5240e0500f64ad9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal\" (UID: \"67f14ea0a754358c5240e0500f64ad9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.746721 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.746701    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.847573 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.847516    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:42.881719 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.881695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.887236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:42.886534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:42.948486 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:42.948454    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:43.049078 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:43.049043    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:43.149668 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:43.149582    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:43.241709 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.241677    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:43.249853 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:43.249827    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:43.272016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.271989    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 11:14:43.272192 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.272174    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 11:14:43.272268 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.272211    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 11:14:43.350561 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:43.350532    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-40.ec2.internal\" not found"
Sep 10 11:14:43.351642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.351619    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 11:09:42 +0000 UTC" deadline="2028-06-02 23:00:28.470995744 +0000 UTC"
Sep 10 11:14:43.351642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.351641    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15155h45m45.119356829s"
Sep 10 11:14:43.409769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.409694    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:43.437675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.437641    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:43.453455 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.453434    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 11:14:43.455187 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.455162    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal"
Sep 10 11:14:43.472329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.472308    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 11:14:43.475408 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:43.475380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f14ea0a754358c5240e0500f64ad9f.slice/crio-9f2ca64b12ae74f2e3a4383858dc94bee81fcbc1adc4a4c263a447b5ac7400e9 WatchSource:0}: Error finding container 9f2ca64b12ae74f2e3a4383858dc94bee81fcbc1adc4a4c263a447b5ac7400e9: Status 404 returned error can't find the container with id 9f2ca64b12ae74f2e3a4383858dc94bee81fcbc1adc4a4c263a447b5ac7400e9
Sep 10 11:14:43.481642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.481625    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:14:43.487457 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:43.487436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6d72dcc0110b2d026681ad98a3f85b.slice/crio-64a3a352904c6bc03f396b09a4dda4e4e7ffc5f6a130c8e61b3691df80ab8a57 WatchSource:0}: Error finding container 64a3a352904c6bc03f396b09a4dda4e4e7ffc5f6a130c8e61b3691df80ab8a57: Status 404 returned error can't find the container with id 64a3a352904c6bc03f396b09a4dda4e4e7ffc5f6a130c8e61b3691df80ab8a57
Sep 10 11:14:43.773627 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:43.773595    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:44.285106 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.285055    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:44.315630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.315333    2578 apiserver.go:52] "Watching apiserver"
Sep 10 11:14:44.326006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.325966    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 11:14:44.326593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.326558    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7","openshift-dns/node-resolver-bbh66","openshift-image-registry/node-ca-cm6c6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal","openshift-multus/multus-xbjg4","openshift-multus/network-metrics-daemon-xmwgb","openshift-network-diagnostics/network-check-target-zczmj","kube-system/global-pull-secret-syncer-dh54m","kube-system/konnectivity-agent-794tn","openshift-cluster-node-tuning-operator/tuned-cm2mq","openshift-multus/multus-additional-cni-plugins-kbbrz","openshift-network-operator/iptables-alerter-x764q","openshift-ovn-kubernetes/ovnkube-node-c24rb"]
Sep 10 11:14:44.329750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.329456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.329750 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.329557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:44.332004 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.331890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.334747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.334363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.336801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.336780    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.337841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.337234    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 11:14:44.337841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.337742    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.338117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.338075    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.338647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.338626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.338796 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.338776    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dd46k\""
Sep 10 11:14:44.338921 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.338903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.339000 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.338985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7wt5d\""
Sep 10 11:14:44.339822 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.339801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.339914 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.339849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 11:14:44.340419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.340398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.340498 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.340476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s2n6s\""
Sep 10 11:14:44.340827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.340805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.343505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.343477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.345235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 11:14:44.345408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vns8z\""
Sep 10 11:14:44.345787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345767    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 11:14:44.345875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.345875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345820    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 11:14:44.345989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345775    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 11:14:44.345989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.345982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.346269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.346237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.346353 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.346314    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:44.347035 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.347014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 11:14:44.347239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.347222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.347420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.347405    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 11:14:44.347493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.347479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6lxq2\""
Sep 10 11:14:44.347554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.347229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.349025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.349005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.351291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-conf\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-node-log\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c3fa39-d36c-470b-99f0-14820da3b3dd-host\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-kubelet\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-systemd-units\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-netd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-config\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-script-lib\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.351929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.352036 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.352021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b13de39-1c19-4465-9e6c-2d077b260bd5-tmp-dir\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.358032 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.352053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65rc\" (UniqueName: \"kubernetes.io/projected/8b13de39-1c19-4465-9e6c-2d077b260bd5-kube-api-access-m65rc\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.360090 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-bin\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.358017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.360224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.358063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:14:44.360353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysconfig\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvj7x\" (UniqueName: \"kubernetes.io/projected/c9441a95-71dc-4f4a-8bce-19c3d088f2be-kube-api-access-pvj7x\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-systemd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-ovn\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.358342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-netns\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-systemd\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.358305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47rvq\""
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-lib-modules\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-var-lib-kubelet\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-tuned\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-tmp\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b13de39-1c19-4465-9e6c-2d077b260bd5-hosts-file\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-log-socket\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-kubernetes\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.360600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-run\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7llh\""
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.359846    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.359920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-sys\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.360919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-etc-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.357977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53846dcd-3283-424b-973c-5a14efcefc95-ovn-node-metrics-cert\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8sc\" (UniqueName: \"kubernetes.io/projected/53846dcd-3283-424b-973c-5a14efcefc95-kube-api-access-vn8sc\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btmf\" (UniqueName: \"kubernetes.io/projected/8b95e13d-b128-4c10-b009-e54830d2be14-kube-api-access-4btmf\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-host\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd44b\" (UniqueName: \"kubernetes.io/projected/85c3fa39-d36c-470b-99f0-14820da3b3dd-kube-api-access-cd44b\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-env-overrides\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.361413    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-modprobe-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.361586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85c3fa39-d36c-470b-99f0-14820da3b3dd-serviceca\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.361837 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-slash\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.361837 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.361627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-var-lib-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.362815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.362783    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 11:14:44.367279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.363297    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 11:14:44.367279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.364544    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 11:14:44.367279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.364676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.367279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.364775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4v54n\""
Sep 10 11:14:44.367279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.365001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 11:14:44.369905 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.369095    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 11:14:44.369905 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.369369    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 11:14:44.369905 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.369545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v28gg\""
Sep 10 11:14:44.401764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.401732    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5v2wb"
Sep 10 11:14:44.417137 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.417107    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5v2wb"
Sep 10 11:14:44.439409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.439382    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 11:14:44.455397 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.455337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal" event={"ID":"3d6d72dcc0110b2d026681ad98a3f85b","Type":"ContainerStarted","Data":"64a3a352904c6bc03f396b09a4dda4e4e7ffc5f6a130c8e61b3691df80ab8a57"}
Sep 10 11:14:44.456801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.456774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal" event={"ID":"67f14ea0a754358c5240e0500f64ad9f","Type":"ContainerStarted","Data":"9f2ca64b12ae74f2e3a4383858dc94bee81fcbc1adc4a4c263a447b5ac7400e9"}
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysconfig\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-systemd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-ovn\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-systemd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-netns\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysconfig\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-systemd\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-ovn\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-tuned\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-tmp\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-run-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-netns\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-systemd\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-konnectivity-ca\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.462497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-multus-certs\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msj8d\" (UniqueName: \"kubernetes.io/projected/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-kube-api-access-msj8d\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-kubernetes\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-run\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-sys\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-host\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nz4n\" (UniqueName: \"kubernetes.io/projected/3a261a5d-76d8-4f04-a290-2ead533c9d34-kube-api-access-5nz4n\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-daemon-config\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-socket-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-k8s-cni-cncf-io\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-node-log\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-netns\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.463252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-kubelet\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-netd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-config\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-script-lib\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.462990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-conf-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-iptables-alerter-script\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-bin\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-registration-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-hostroot\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvj7x\" (UniqueName: \"kubernetes.io/projected/c9441a95-71dc-4f4a-8bce-19c3d088f2be-kube-api-access-pvj7x\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-sys-fs\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cnibin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cni-binary-copy\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-socket-dir-parent\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-bin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-lib-modules\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-var-lib-kubelet\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b13de39-1c19-4465-9e6c-2d077b260bd5-hosts-file\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-netd\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-log-socket\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-log-socket\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-host\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-node-log\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.463789    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-kubelet\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.463548    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.463875    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:14:44.963837054 +0000 UTC m=+3.039162715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-cni-bin\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-sys\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.464660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8sc\" (UniqueName: \"kubernetes.io/projected/53846dcd-3283-424b-973c-5a14efcefc95-kube-api-access-vn8sc\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btmf\" (UniqueName: \"kubernetes.io/projected/8b95e13d-b128-4c10-b009-e54830d2be14-kube-api-access-4btmf\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-device-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-etc-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53846dcd-3283-424b-973c-5a14efcefc95-ovn-node-metrics-cert\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-run\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-run-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-config\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-var-lib-kubelet\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b13de39-1c19-4465-9e6c-2d077b260bd5-hosts-file\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-lib-modules\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-ovnkube-script-lib\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-etc-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-kubernetes\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-dbus\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.464983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-etc-selinux\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.465669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-cnibin\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds5q\" (UniqueName: \"kubernetes.io/projected/18362d46-4341-4ae6-b491-19876425251e-kube-api-access-tds5q\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-system-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd44b\" (UniqueName: \"kubernetes.io/projected/85c3fa39-d36c-470b-99f0-14820da3b3dd-kube-api-access-cd44b\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-env-overrides\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-kubelet\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-modprobe-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85c3fa39-d36c-470b-99f0-14820da3b3dd-serviceca\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-slash\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-slash\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-modprobe-d\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-var-lib-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-multus\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-var-lib-openvswitch\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-conf\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-etc-kubernetes\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-host-slash\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.466722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-kubelet-config\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-sysctl-conf\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c3fa39-d36c-470b-99f0-14820da3b3dd-host\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53846dcd-3283-424b-973c-5a14efcefc95-env-overrides\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-systemd-units\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c3fa39-d36c-470b-99f0-14820da3b3dd-host\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-agent-certs\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-systemd-units\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b13de39-1c19-4465-9e6c-2d077b260bd5-tmp-dir\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65rc\" (UniqueName: \"kubernetes.io/projected/8b13de39-1c19-4465-9e6c-2d077b260bd5-kube-api-access-m65rc\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85c3fa39-d36c-470b-99f0-14820da3b3dd-serviceca\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.465912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn24d\" (UniqueName: \"kubernetes.io/projected/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-kube-api-access-bn24d\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.466035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53846dcd-3283-424b-973c-5a14efcefc95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.467670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.466100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b13de39-1c19-4465-9e6c-2d077b260bd5-tmp-dir\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.468374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.468192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-tmp\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.468374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.468215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9441a95-71dc-4f4a-8bce-19c3d088f2be-etc-tuned\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.469065 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.469043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53846dcd-3283-424b-973c-5a14efcefc95-ovn-node-metrics-cert\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.489210 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.488576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvj7x\" (UniqueName: \"kubernetes.io/projected/c9441a95-71dc-4f4a-8bce-19c3d088f2be-kube-api-access-pvj7x\") pod \"tuned-cm2mq\" (UID: \"c9441a95-71dc-4f4a-8bce-19c3d088f2be\") " pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.500375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.500339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd44b\" (UniqueName: \"kubernetes.io/projected/85c3fa39-d36c-470b-99f0-14820da3b3dd-kube-api-access-cd44b\") pod \"node-ca-cm6c6\" (UID: \"85c3fa39-d36c-470b-99f0-14820da3b3dd\") " pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.503705 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.503665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btmf\" (UniqueName: \"kubernetes.io/projected/8b95e13d-b128-4c10-b009-e54830d2be14-kube-api-access-4btmf\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.504389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.504360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65rc\" (UniqueName: \"kubernetes.io/projected/8b13de39-1c19-4465-9e6c-2d077b260bd5-kube-api-access-m65rc\") pod \"node-resolver-bbh66\" (UID: \"8b13de39-1c19-4465-9e6c-2d077b260bd5\") " pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.506665 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.506629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8sc\" (UniqueName: \"kubernetes.io/projected/53846dcd-3283-424b-973c-5a14efcefc95-kube-api-access-vn8sc\") pod \"ovnkube-node-c24rb\" (UID: \"53846dcd-3283-424b-973c-5a14efcefc95\") " pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.569335 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nz4n\" (UniqueName: \"kubernetes.io/projected/3a261a5d-76d8-4f04-a290-2ead533c9d34-kube-api-access-5nz4n\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.569335 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-daemon-config\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-socket-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-k8s-cni-cncf-io\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-netns\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569569 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-conf-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-iptables-alerter-script\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-registration-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-hostroot\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-sys-fs\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cnibin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cni-binary-copy\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-socket-dir-parent\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.569915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-bin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-device-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-dbus\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.569996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-etc-selinux\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-cnibin\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds5q\" (UniqueName: \"kubernetes.io/projected/18362d46-4341-4ae6-b491-19876425251e-kube-api-access-tds5q\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-system-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-kubelet\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-multus\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-etc-kubernetes\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-host-slash\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-bin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-kubelet-config\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-k8s-cni-cncf-io\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-sys-fs\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-kubelet-config\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-agent-certs\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.570538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-socket-dir-parent\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn24d\" (UniqueName: \"kubernetes.io/projected/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-kube-api-access-bn24d\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-konnectivity-ca\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-etc-selinux\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-socket-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-device-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-cnibin\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-system-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-kubelet\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cni-binary-copy\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-host-slash\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.570998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-hostroot\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-registration-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-var-lib-cni-multus\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.571409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-konnectivity-ca\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a261a5d-76d8-4f04-a290-2ead533c9d34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34887360-7db4-44aa-8c7a-976d77303b5b-dbus\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.571337    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.571412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:14:45.071390764 +0000 UTC m=+3.146716424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-conf-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-multus-certs\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msj8d\" (UniqueName: \"kubernetes.io/projected/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-kube-api-access-msj8d\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-iptables-alerter-script\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18362d46-4341-4ae6-b491-19876425251e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-netns\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-host-run-multus-certs\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-cni-dir\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.571983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-etc-kubernetes\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.572103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-cnibin\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.572174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-multus-daemon-config\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.572701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.572244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.572701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.572568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18362d46-4341-4ae6-b491-19876425251e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.575692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.575662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb-agent-certs\") pod \"konnectivity-agent-794tn\" (UID: \"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb\") " pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.601656 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.601616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn24d\" (UniqueName: \"kubernetes.io/projected/f8b6f48b-17e0-40fd-bc66-67abb3f0593f-kube-api-access-bn24d\") pod \"iptables-alerter-x764q\" (UID: \"f8b6f48b-17e0-40fd-bc66-67abb3f0593f\") " pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.601853 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.601829    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:14:44.601950 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.601859    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:14:44.601950 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.601876    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:44.602066 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.601957    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:14:45.101938463 +0000 UTC m=+3.177264129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:44.604064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.604016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nz4n\" (UniqueName: \"kubernetes.io/projected/3a261a5d-76d8-4f04-a290-2ead533c9d34-kube-api-access-5nz4n\") pod \"aws-ebs-csi-driver-node-c84r7\" (UID: \"3a261a5d-76d8-4f04-a290-2ead533c9d34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.604940 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.604839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds5q\" (UniqueName: \"kubernetes.io/projected/18362d46-4341-4ae6-b491-19876425251e-kube-api-access-tds5q\") pod \"multus-additional-cni-plugins-kbbrz\" (UID: \"18362d46-4341-4ae6-b491-19876425251e\") " pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.606974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.606953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msj8d\" (UniqueName: \"kubernetes.io/projected/82669b41-fda9-4b4c-b30f-c1c4ed5d1733-kube-api-access-msj8d\") pod \"multus-xbjg4\" (UID: \"82669b41-fda9-4b4c-b30f-c1c4ed5d1733\") " pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.648021 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.647725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cm2mq"
Sep 10 11:14:44.664320 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.664282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bbh66"
Sep 10 11:14:44.664697 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:44.664664    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9441a95_71dc_4f4a_8bce_19c3d088f2be.slice/crio-65042ebc1d8abe098885970d2048f3195c3bfeed49242c6125e1c958a1c37588 WatchSource:0}: Error finding container 65042ebc1d8abe098885970d2048f3195c3bfeed49242c6125e1c958a1c37588: Status 404 returned error can't find the container with id 65042ebc1d8abe098885970d2048f3195c3bfeed49242c6125e1c958a1c37588
Sep 10 11:14:44.675604 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.675577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm6c6"
Sep 10 11:14:44.681409 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:44.681379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b13de39_1c19_4465_9e6c_2d077b260bd5.slice/crio-120c0330e6d1ffa3bde157f1fa1e8fffd40c540a5f527bbf4fcb8a5384ae7dba WatchSource:0}: Error finding container 120c0330e6d1ffa3bde157f1fa1e8fffd40c540a5f527bbf4fcb8a5384ae7dba: Status 404 returned error can't find the container with id 120c0330e6d1ffa3bde157f1fa1e8fffd40c540a5f527bbf4fcb8a5384ae7dba
Sep 10 11:14:44.682308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.682290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:14:44.685552 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:44.685530    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c3fa39_d36c_470b_99f0_14820da3b3dd.slice/crio-6f8333564c2fb22e81bc78d8d60c0b2e7f08dac193cb589f873f644b934bc5f5 WatchSource:0}: Error finding container 6f8333564c2fb22e81bc78d8d60c0b2e7f08dac193cb589f873f644b934bc5f5: Status 404 returned error can't find the container with id 6f8333564c2fb22e81bc78d8d60c0b2e7f08dac193cb589f873f644b934bc5f5
Sep 10 11:14:44.690987 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:44.690962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53846dcd_3283_424b_973c_5a14efcefc95.slice/crio-575ca9a413ec8743a06d2f121a32f33d07afe8829d52b6b357c11506fa86170c WatchSource:0}: Error finding container 575ca9a413ec8743a06d2f121a32f33d07afe8829d52b6b357c11506fa86170c: Status 404 returned error can't find the container with id 575ca9a413ec8743a06d2f121a32f33d07afe8829d52b6b357c11506fa86170c
Sep 10 11:14:44.691069 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.691032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbjg4"
Sep 10 11:14:44.699187 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:44.699141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82669b41_fda9_4b4c_b30f_c1c4ed5d1733.slice/crio-ff0a9a1aff49938803925fb7e1fc5e8cf6dee3f8f1e7c4ac9c7a4fce464ce00f WatchSource:0}: Error finding container ff0a9a1aff49938803925fb7e1fc5e8cf6dee3f8f1e7c4ac9c7a4fce464ce00f: Status 404 returned error can't find the container with id ff0a9a1aff49938803925fb7e1fc5e8cf6dee3f8f1e7c4ac9c7a4fce464ce00f
Sep 10 11:14:44.700198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.700178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x764q"
Sep 10 11:14:44.710054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.710032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7"
Sep 10 11:14:44.719056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.719035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:14:44.726223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.726199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbbrz"
Sep 10 11:14:44.975335 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:44.975298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:44.975545 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.975489    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:44.975600 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:44.975579    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:14:45.975557143 +0000 UTC m=+4.050882796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:45.049513 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:45.049466    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b6f48b_17e0_40fd_bc66_67abb3f0593f.slice/crio-68e873add88ddcb748800cd21752186e4cdbcc291024e615ad7c12f2bcb1088c WatchSource:0}: Error finding container 68e873add88ddcb748800cd21752186e4cdbcc291024e615ad7c12f2bcb1088c: Status 404 returned error can't find the container with id 68e873add88ddcb748800cd21752186e4cdbcc291024e615ad7c12f2bcb1088c
Sep 10 11:14:45.076271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.076234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:45.076415 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.076385    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:45.076477 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.076455    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:14:46.076437611 +0000 UTC m=+4.151763255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:45.177163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.177120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:45.177362 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.177265    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:14:45.177362 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.177292    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:14:45.177362 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.177308    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:45.177535 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.177381    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:14:46.177361836 +0000 UTC m=+4.252687485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:45.420949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.420888    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:09:44 +0000 UTC" deadline="2028-06-12 01:28:58.91432403 +0000 UTC"
Sep 10 11:14:45.420949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.420947    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15374h14m13.493382962s"
Sep 10 11:14:45.449654 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.449616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:45.449820 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.449795    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:45.460637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.460597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm6c6" event={"ID":"85c3fa39-d36c-470b-99f0-14820da3b3dd","Type":"ContainerStarted","Data":"6f8333564c2fb22e81bc78d8d60c0b2e7f08dac193cb589f873f644b934bc5f5"}
Sep 10 11:14:45.461973 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.461941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bbh66" event={"ID":"8b13de39-1c19-4465-9e6c-2d077b260bd5","Type":"ContainerStarted","Data":"120c0330e6d1ffa3bde157f1fa1e8fffd40c540a5f527bbf4fcb8a5384ae7dba"}
Sep 10 11:14:45.463336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.463303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cm2mq" event={"ID":"c9441a95-71dc-4f4a-8bce-19c3d088f2be","Type":"ContainerStarted","Data":"65042ebc1d8abe098885970d2048f3195c3bfeed49242c6125e1c958a1c37588"}
Sep 10 11:14:45.464550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.464527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x764q" event={"ID":"f8b6f48b-17e0-40fd-bc66-67abb3f0593f","Type":"ContainerStarted","Data":"68e873add88ddcb748800cd21752186e4cdbcc291024e615ad7c12f2bcb1088c"}
Sep 10 11:14:45.465752 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.465729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbjg4" event={"ID":"82669b41-fda9-4b4c-b30f-c1c4ed5d1733","Type":"ContainerStarted","Data":"ff0a9a1aff49938803925fb7e1fc5e8cf6dee3f8f1e7c4ac9c7a4fce464ce00f"}
Sep 10 11:14:45.467164 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.467138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"575ca9a413ec8743a06d2f121a32f33d07afe8829d52b6b357c11506fa86170c"}
Sep 10 11:14:45.875755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.875720    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:14:45.983648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:45.983603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:45.983821 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.983787    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:45.983867 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:45.983852    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:14:47.983833493 +0000 UTC m=+6.059159140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:46.084331 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.084277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:46.084510 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.084456    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:46.084570 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.084520    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:14:48.084501957 +0000 UTC m=+6.159827615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:46.185232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.185146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:46.185405 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.185296    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:14:46.185405 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.185313    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:14:46.185405 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.185324    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:46.185405 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.185369    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:14:48.185356808 +0000 UTC m=+6.260682452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:46.246621 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:46.246586    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa45f9c_d6a7_4a89_b4df_4048ea79d0eb.slice/crio-78722d0a9cbbb0e8f80278f832a1abe0dcff9e7bb8a1e827906395a9e48bb344 WatchSource:0}: Error finding container 78722d0a9cbbb0e8f80278f832a1abe0dcff9e7bb8a1e827906395a9e48bb344: Status 404 returned error can't find the container with id 78722d0a9cbbb0e8f80278f832a1abe0dcff9e7bb8a1e827906395a9e48bb344
Sep 10 11:14:46.266831 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:14:46.266487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18362d46_4341_4ae6_b491_19876425251e.slice/crio-89c2eaf34bd45d51a6556a9371c27d533989aa4ecb5ad1920411331286abea8a WatchSource:0}: Error finding container 89c2eaf34bd45d51a6556a9371c27d533989aa4ecb5ad1920411331286abea8a: Status 404 returned error can't find the container with id 89c2eaf34bd45d51a6556a9371c27d533989aa4ecb5ad1920411331286abea8a
Sep 10 11:14:46.428209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.427738    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:09:44 +0000 UTC" deadline="2028-04-03 20:55:53.300737017 +0000 UTC"
Sep 10 11:14:46.428209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.428023    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13713h41m6.87272142s"
Sep 10 11:14:46.449812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.449701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:46.449964 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.449828    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:46.449964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.449882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:46.449964 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:46.449938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:46.474337 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.474263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerStarted","Data":"89c2eaf34bd45d51a6556a9371c27d533989aa4ecb5ad1920411331286abea8a"}
Sep 10 11:14:46.477488 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.477395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7" event={"ID":"3a261a5d-76d8-4f04-a290-2ead533c9d34","Type":"ContainerStarted","Data":"c03ad77d64cc6095871acd2f80fd730138bea5638d5c59ca9cc941208d217764"}
Sep 10 11:14:46.480606 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.480568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-794tn" event={"ID":"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb","Type":"ContainerStarted","Data":"78722d0a9cbbb0e8f80278f832a1abe0dcff9e7bb8a1e827906395a9e48bb344"}
Sep 10 11:14:46.485704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.485666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal" event={"ID":"3d6d72dcc0110b2d026681ad98a3f85b","Type":"ContainerStarted","Data":"5c467862fbcd97c96d987d82df8892d1d5f155df6d3aca82f6b5259b48211309"}
Sep 10 11:14:46.511735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:46.511665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-40.ec2.internal" podStartSLOduration=3.511644446 podStartE2EDuration="3.511644446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:14:46.510849391 +0000 UTC m=+4.586175054" watchObservedRunningTime="2026-09-10 11:14:46.511644446 +0000 UTC m=+4.586970135"
Sep 10 11:14:47.450097 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:47.450047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:47.450549 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:47.450220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:47.491201 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:47.491143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal" event={"ID":"67f14ea0a754358c5240e0500f64ad9f","Type":"ContainerStarted","Data":"573b402cf320e9900444b6d13c972d0d26b46764a0ebca3d8eb72c5b73b030d7"}
Sep 10 11:14:48.002522 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:48.002482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:48.002722 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.002632    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:48.002722 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.002706    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:14:52.002685256 +0000 UTC m=+10.078010903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:48.103024 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:48.102981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:48.103269 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.103192    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:48.103269 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.103262    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:14:52.103245475 +0000 UTC m=+10.178571132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:48.203950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:48.203899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:48.204159 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.204099    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:14:48.204159 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.204123    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:14:48.204159 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.204136    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:48.204353 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.204202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:14:52.204182906 +0000 UTC m=+10.279508571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:48.450682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:48.450186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:48.450682 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.450314    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:48.451503 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:48.451325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:48.451503 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:48.451429    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:49.449745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:49.449701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:49.449902 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:49.449880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:49.507769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:49.507732    2578 generic.go:356] "Generic (PLEG): container finished" podID="67f14ea0a754358c5240e0500f64ad9f" containerID="573b402cf320e9900444b6d13c972d0d26b46764a0ebca3d8eb72c5b73b030d7" exitCode=0
Sep 10 11:14:49.508234 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:49.507785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal" event={"ID":"67f14ea0a754358c5240e0500f64ad9f","Type":"ContainerDied","Data":"573b402cf320e9900444b6d13c972d0d26b46764a0ebca3d8eb72c5b73b030d7"}
Sep 10 11:14:50.452398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:50.452263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:50.452572 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:50.452398    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:50.452769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:50.452751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:50.452853 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:50.452838    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:51.450147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:51.450110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:51.450614 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:51.450250    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:52.037574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:52.037530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:52.037761 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.037706    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:52.037821 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.037767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:00.037749386 +0000 UTC m=+18.113075044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:14:52.138624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:52.138584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:52.138793 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.138756    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:52.138866 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.138830    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:00.138810817 +0000 UTC m=+18.214136472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:14:52.239937 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:52.239894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:52.240134 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.240105    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:14:52.240134 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.240130    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:14:52.240248 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.240144    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:52.240248 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.240204    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:00.240184723 +0000 UTC m=+18.315510384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:14:52.450285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:52.450251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:52.450720 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.450370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:52.450720 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:52.450414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:52.450720 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:52.450511    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:53.449880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:53.449839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:53.450076 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:53.449979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:54.449735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:54.449695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:54.449735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:54.449712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:54.450334 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:54.449831    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:54.450334 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:54.449948    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:55.449848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:55.449817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:55.450229 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:55.449919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:56.450352 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:56.450317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:56.450814 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:56.450457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:56.450814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:56.450513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:56.450814 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:56.450626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:57.449745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:57.449703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:57.449915 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:57.449876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:14:58.449640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:58.449598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:14:58.450076 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:58.449648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:14:58.450076 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:58.449740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:14:58.450076 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:58.449856    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:14:59.450141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:14:59.450105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:14:59.450624 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:14:59.450220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:00.099719 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:00.099682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:00.099906 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.099819    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:15:00.099906 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.099888    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:16.099871143 +0000 UTC m=+34.175196785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:15:00.200917 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:00.200875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:00.201111 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.201002    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:15:00.201111 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.201063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:16.20104284 +0000 UTC m=+34.276368489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:15:00.301594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:00.301549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:00.301764 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.301707    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:15:00.301764 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.301723    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:15:00.301764 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.301732    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:15:00.301894 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.301796    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:16.301779872 +0000 UTC m=+34.377105517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:15:00.450216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:00.450179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:00.450629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:00.450177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:00.450629 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.450316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:00.450629 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:00.450360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:01.449820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:01.449783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:01.450014 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:01.449936    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:02.450676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:02.450640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:02.451098 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:02.450716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:02.451098 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:02.450732    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:02.451098 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:02.450808    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:03.449473 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.449322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:03.449610 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:03.449554    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:03.539047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.539018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bbh66" event={"ID":"8b13de39-1c19-4465-9e6c-2d077b260bd5","Type":"ContainerStarted","Data":"524d24531e07da663f7b05238d47b8969a40a2a4fb6f5dc2b36e53a46438553e"}
Sep 10 11:15:03.540781 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.540756    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="6e804809874670befa5cbf29b28efcef6615db584c0dc9395c1aedaeed738627" exitCode=0
Sep 10 11:15:03.540902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.540835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"6e804809874670befa5cbf29b28efcef6615db584c0dc9395c1aedaeed738627"}
Sep 10 11:15:03.546703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.546676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal" event={"ID":"67f14ea0a754358c5240e0500f64ad9f","Type":"ContainerStarted","Data":"8f74bf05299cae73a82704f9a67aed48d25ab67265ffdc8226742edc5843dfcb"}
Sep 10 11:15:03.548136 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.548107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm6c6" event={"ID":"85c3fa39-d36c-470b-99f0-14820da3b3dd","Type":"ContainerStarted","Data":"b8cf409e43b1ddadaac945f52ef5a0c842386a506dd1927946043a1ad562b10f"}
Sep 10 11:15:03.549446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.549428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cm2mq" event={"ID":"c9441a95-71dc-4f4a-8bce-19c3d088f2be","Type":"ContainerStarted","Data":"f7e9069ce13508c9b864b5bb51fffe935fd0507192a99db0e325bf65a35273d3"}
Sep 10 11:15:03.550892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.550872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7" event={"ID":"3a261a5d-76d8-4f04-a290-2ead533c9d34","Type":"ContainerStarted","Data":"2cfc229a0b1f9c993940d361db1c083cbc076bfbfa2f7d0dd59339a9f7184ab9"}
Sep 10 11:15:03.552241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.552222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-794tn" event={"ID":"aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb","Type":"ContainerStarted","Data":"0a28ec61f425578aa10ae46f385442b3963cf914338b3919297bb3297632309d"}
Sep 10 11:15:03.553674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.553649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbjg4" event={"ID":"82669b41-fda9-4b4c-b30f-c1c4ed5d1733","Type":"ContainerStarted","Data":"dcf6531c9d01c0e1b68ebeac5d49a90b4b63920e9d728833bf77d045295000d2"}
Sep 10 11:15:03.555774 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.555752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"4103f4eaa805842d506e010575b9c64f55ec5310f48629277a1bec08b2d73a6c"}
Sep 10 11:15:03.555876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.555830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"f962ca3115ab870c6213ef889e322e21363f5ceabaa1dfa4969dd0316eb5e6e6"}
Sep 10 11:15:03.555918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.555881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"e718b6f881def2b6f78fb2a54cbcc82178a6aaab47b59b20f175c27e4fea9b2d"}
Sep 10 11:15:03.595586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.595549    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bbh66" podStartSLOduration=7.387628851 podStartE2EDuration="21.595534452s" totalImagesPullingTime="14.207905601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.569061808 +0000 UTC m=+21.644387469" watchObservedRunningTime="2026-09-10 11:15:03.595534452 +0000 UTC m=+21.670860117"
Sep 10 11:15:03.627836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.627797    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-794tn" podStartSLOduration=4.80936066 podStartE2EDuration="21.627783628s" totalImagesPullingTime="16.818422968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.596631033 +0000 UTC m=+21.671956694" watchObservedRunningTime="2026-09-10 11:15:03.627783628 +0000 UTC m=+21.703109292"
Sep 10 11:15:03.677136 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.677061    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-40.ec2.internal" podStartSLOduration=20.677047854 podStartE2EDuration="20.677047854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.676883116 +0000 UTC m=+21.752208779" watchObservedRunningTime="2026-09-10 11:15:03.677047854 +0000 UTC m=+21.752373517"
Sep 10 11:15:03.715157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.715119    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cm2mq" podStartSLOduration=3.312449929 podStartE2EDuration="21.71510569s" totalImagesPullingTime="18.402655761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.714613852 +0000 UTC m=+21.789939516" watchObservedRunningTime="2026-09-10 11:15:03.71510569 +0000 UTC m=+21.790431348"
Sep 10 11:15:03.760292 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.760251    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cm6c6" podStartSLOduration=3.379425112 podStartE2EDuration="21.760235147s" totalImagesPullingTime="18.380810035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.757492446 +0000 UTC m=+21.832818122" watchObservedRunningTime="2026-09-10 11:15:03.760235147 +0000 UTC m=+21.835560789"
Sep 10 11:15:03.786532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.786491    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xbjg4" podStartSLOduration=3.368119377 podStartE2EDuration="21.786475107s" totalImagesPullingTime="18.41835573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:03.786474617 +0000 UTC m=+21.861800281" watchObservedRunningTime="2026-09-10 11:15:03.786475107 +0000 UTC m=+21.861800771"
Sep 10 11:15:03.960685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:03.960613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:04.451836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.451797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:04.452112 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:04.451943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:04.452112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.451976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:04.452198 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:04.452113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:04.501142 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.501105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:04.560208 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.560170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x764q" event={"ID":"f8b6f48b-17e0-40fd-bc66-67abb3f0593f","Type":"ContainerStarted","Data":"762bfa248747e7b2748f1e9eadeb4963215ef9540265526e9a075d51d68359c8"}
Sep 10 11:15:04.564412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.564381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"6f90309932f8e603c1bd55e776d455604665750942a894eebeaa36a2fae0e130"}
Sep 10 11:15:04.564552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.564466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"bae9581fe7a3cecdc3ec3def125ee92dc369704af385bae3cd525ad80b971f75"}
Sep 10 11:15:04.564552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.564513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"5e87d52a72ee3beb92cf53e287a617cc85439a8c9db5d01a19b029fd052581ae"}
Sep 10 11:15:04.591526 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.588455    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x764q" podStartSLOduration=4.56700603 podStartE2EDuration="22.588436668s" totalImagesPullingTime="18.021430638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:04.587404789 +0000 UTC m=+22.662730449" watchObservedRunningTime="2026-09-10 11:15:04.588436668 +0000 UTC m=+22.663762359"
Sep 10 11:15:04.734333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:04.734307    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 11:15:05.417369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:05.417250    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T11:15:04.734329963Z","UUID":"266effd7-62e8-46e5-96c1-4dd6066a9428","Handler":null,"Name":"","Endpoint":""}
Sep 10 11:15:05.419117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:05.419074    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 11:15:05.419117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:05.419130    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 11:15:05.450439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:05.450409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:05.450614 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:05.450552    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:05.569480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:05.569441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7" event={"ID":"3a261a5d-76d8-4f04-a290-2ead533c9d34","Type":"ContainerStarted","Data":"9c726525c327ad8f9c1467223d11538325aa90c246f36a6878b5693859a013f1"}
Sep 10 11:15:06.180318 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.180112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:06.331671 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.331632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:06.449692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.449615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:06.449692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.449649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:06.449892 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:06.449734    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:06.449892 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:06.449844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:06.573820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.573783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7" event={"ID":"3a261a5d-76d8-4f04-a290-2ead533c9d34","Type":"ContainerStarted","Data":"44e7f808d27a3c0ab53d05f3f557902280d9d03dc8805cee96b68708096dca3a"}
Sep 10 11:15:06.577746 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.577707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"236e3e1b6368b8f1389228c246b5d64c727de0c31bcdbef0a8d701aea7ef8936"}
Sep 10 11:15:06.613389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.613335    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c84r7" podStartSLOduration=5.499956121 podStartE2EDuration="24.613312244s" totalImagesPullingTime="19.113356123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:06.609855153 +0000 UTC m=+24.685180818" watchObservedRunningTime="2026-09-10 11:15:06.613312244 +0000 UTC m=+24.688638093"
Sep 10 11:15:06.796403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.796315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:15:06.797005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:06.796984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:15:07.450133 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:07.450096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:07.450355 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:07.450241    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:07.580644 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:07.580613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:15:07.581199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:07.581161    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-794tn"
Sep 10 11:15:07.750061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:07.749987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:08.449948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.449764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:08.450172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.449829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:08.450172 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:08.450039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:08.450172 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:08.450136    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:08.586298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.586255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" event={"ID":"53846dcd-3283-424b-973c-5a14efcefc95","Type":"ContainerStarted","Data":"23ffd7a4b1142b8106e4a2c4a1fb50d2b1997ae168d5b3bc5e2940fb1fe35e69"}
Sep 10 11:15:08.587068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.586548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:08.588114 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.588073    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="f9ef6332613db8b10ce3dce86f693bba2bb555362ad090648e89dd307f8222a6" exitCode=0
Sep 10 11:15:08.588221 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.588163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"f9ef6332613db8b10ce3dce86f693bba2bb555362ad090648e89dd307f8222a6"}
Sep 10 11:15:08.603156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.603137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:08.630076 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:08.630027    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb" podStartSLOduration=8.147542154 podStartE2EDuration="26.630013174s" totalImagesPullingTime="18.48247102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:08.628336231 +0000 UTC m=+26.703661895" watchObservedRunningTime="2026-09-10 11:15:08.630013174 +0000 UTC m=+26.705338857"
Sep 10 11:15:09.449634 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.449613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:09.449741 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:09.449720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:09.593701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.593661    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="e6b361f80fe979f9c054505da8b31a71f54ece72f09660a2639e07a2cde42032" exitCode=0
Sep 10 11:15:09.594149 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.593775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"e6b361f80fe979f9c054505da8b31a71f54ece72f09660a2639e07a2cde42032"}
Sep 10 11:15:09.601625 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.601594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:09.601728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.601708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:09.624487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:09.624397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:10.449641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.449462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:10.449821 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.449531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:10.449821 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:10.449731    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:10.449903 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:10.449849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:10.501151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.501038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmwgb"]
Sep 10 11:15:10.501317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.501205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:10.501382 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:10.501321    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:10.501954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.501936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zczmj"]
Sep 10 11:15:10.550837 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.550798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh54m"]
Sep 10 11:15:10.598668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.598631    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="637a9c26b4c682d6e19dfe13e88bbdb55643414149d891a7b39866ea30819f1f" exitCode=0
Sep 10 11:15:10.599115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.598744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"637a9c26b4c682d6e19dfe13e88bbdb55643414149d891a7b39866ea30819f1f"}
Sep 10 11:15:10.599115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.598764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:10.599200 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:10.599147    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:10.599200 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.599171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:10.599788 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:10.599285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:10.678497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:10.678441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:11.168525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:11.168487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:11.281348 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:11.281316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:12.450808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:12.450775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:12.451301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:12.450829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:12.451301 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:12.450896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:12.451301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:12.450928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:12.451301 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:12.451031    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:12.451301 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:12.451134    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:14.449700 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:14.449660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:14.450418 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:14.449793    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:14.450418 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:14.450241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:14.450418 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:14.450370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:14.450586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:14.450487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:14.450586 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:14.450562    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:15.154009 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:15.153957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:15.237027 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:15.236992    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:16.059072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.059035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:16.121187 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.121159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:16.121345 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.121303    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:15:16.121385 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.121367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:48.121349472 +0000 UTC m=+66.196675129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:15:16.166261 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.166228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:16.221767 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.221730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:16.221929 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.221829    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:15:16.221929 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.221880    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret podName:34887360-7db4-44aa-8c7a-976d77303b5b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:48.221867253 +0000 UTC m=+66.297192895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret") pod "global-pull-secret-syncer-dh54m" (UID: "34887360-7db4-44aa-8c7a-976d77303b5b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:15:16.322596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.322516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:16.322751 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.322690    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:15:16.322751 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.322715    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:15:16.322751 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.322724    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mvnpr for pod openshift-network-diagnostics/network-check-target-zczmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:15:16.322874 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.322773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr podName:9e0341b5-bfe5-4b77-a855-9fd30af1817f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:48.3227597 +0000 UTC m=+66.398085342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvnpr" (UniqueName: "kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr") pod "network-check-target-zczmj" (UID: "9e0341b5-bfe5-4b77-a855-9fd30af1817f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:15:16.450342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.450318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:16.450433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.450327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:16.450433 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.450416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zczmj" podUID="9e0341b5-bfe5-4b77-a855-9fd30af1817f"
Sep 10 11:15:16.450531 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.450510    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh54m" podUID="34887360-7db4-44aa-8c7a-976d77303b5b"
Sep 10 11:15:16.450567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.450544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:16.450616 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:16.450604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmwgb" podUID="8b95e13d-b128-4c10-b009-e54830d2be14"
Sep 10 11:15:16.617745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.617715    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="52f6f793fc1f3f03593d304c9952cd0b2d507fe9fa7d29d86d00b17c9acabde2" exitCode=0
Sep 10 11:15:16.617875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.617792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"52f6f793fc1f3f03593d304c9952cd0b2d507fe9fa7d29d86d00b17c9acabde2"}
Sep 10 11:15:16.713228 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.713206    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-40.ec2.internal" event="NodeReady"
Sep 10 11:15:16.713354 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.713343    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 11:15:16.825412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.825384    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6tq59"]
Sep 10 11:15:16.848517 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.848448    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbfxf"]
Sep 10 11:15:16.848644 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.848561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:16.852027 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.852002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 11:15:16.852187 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.852119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 11:15:16.852396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.852374    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 11:15:16.852747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.852731    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w7x6j\""
Sep 10 11:15:16.862493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.862473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tq59"]
Sep 10 11:15:16.862583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.862532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:16.868872 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.868853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 11:15:16.869139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.869125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-krm6r\""
Sep 10 11:15:16.870767 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.870752    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 11:15:16.879076 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:16.879051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbfxf"]
Sep 10 11:15:17.018243 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.018209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:17.027217 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493c32a-781d-4ced-9053-f22d34da37af-cert\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.027305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.027305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdgf\" (UniqueName: \"kubernetes.io/projected/5e3b4414-485e-4bf7-8743-c0b0b6a47226-kube-api-access-qbdgf\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.027305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3b4414-485e-4bf7-8743-c0b0b6a47226-tmp-dir\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.027416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3b4414-485e-4bf7-8743-c0b0b6a47226-config-volume\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.027416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.027378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2lz\" (UniqueName: \"kubernetes.io/projected/c493c32a-781d-4ced-9053-f22d34da37af-kube-api-access-bl2lz\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.128594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.128560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.129155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.128600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdgf\" (UniqueName: \"kubernetes.io/projected/5e3b4414-485e-4bf7-8743-c0b0b6a47226-kube-api-access-qbdgf\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.129155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.128634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3b4414-485e-4bf7-8743-c0b0b6a47226-tmp-dir\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.129155 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:17.128712    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:17.129155 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:17.128767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:17.628752054 +0000 UTC m=+35.704077696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:17.129155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.128807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3b4414-485e-4bf7-8743-c0b0b6a47226-config-volume\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.129466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.129168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2lz\" (UniqueName: \"kubernetes.io/projected/c493c32a-781d-4ced-9053-f22d34da37af-kube-api-access-bl2lz\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.129466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.129209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493c32a-781d-4ced-9053-f22d34da37af-cert\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.129466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.129369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3b4414-485e-4bf7-8743-c0b0b6a47226-tmp-dir\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.129779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.129754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3b4414-485e-4bf7-8743-c0b0b6a47226-config-volume\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.134064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.134041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493c32a-781d-4ced-9053-f22d34da37af-cert\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.148233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.148214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdgf\" (UniqueName: \"kubernetes.io/projected/5e3b4414-485e-4bf7-8743-c0b0b6a47226-kube-api-access-qbdgf\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.152075 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.152058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2lz\" (UniqueName: \"kubernetes.io/projected/c493c32a-781d-4ced-9053-f22d34da37af-kube-api-access-bl2lz\") pod \"ingress-canary-6tq59\" (UID: \"c493c32a-781d-4ced-9053-f22d34da37af\") " pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.167844 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.167818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tq59"
Sep 10 11:15:17.243620 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.243449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:17.502223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.502075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tq59"]
Sep 10 11:15:17.506593 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:17.506564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493c32a_781d_4ced_9053_f22d34da37af.slice/crio-dfa2d636566acb84bd59622097a09117089195109e33e041acb9f24d43c96cda WatchSource:0}: Error finding container dfa2d636566acb84bd59622097a09117089195109e33e041acb9f24d43c96cda: Status 404 returned error can't find the container with id dfa2d636566acb84bd59622097a09117089195109e33e041acb9f24d43c96cda
Sep 10 11:15:17.623212 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.623177    2578 generic.go:356] "Generic (PLEG): container finished" podID="18362d46-4341-4ae6-b491-19876425251e" containerID="c8d11b66250667084b08edc4085edd359b4328af655b9c48f5f6c17997064e44" exitCode=0
Sep 10 11:15:17.623425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.623249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerDied","Data":"c8d11b66250667084b08edc4085edd359b4328af655b9c48f5f6c17997064e44"}
Sep 10 11:15:17.624488 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.624464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tq59" event={"ID":"c493c32a-781d-4ced-9053-f22d34da37af","Type":"ContainerStarted","Data":"dfa2d636566acb84bd59622097a09117089195109e33e041acb9f24d43c96cda"}
Sep 10 11:15:17.634170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.634152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:17.634261 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:17.634248    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:17.634308 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:17.634298    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:18.63428258 +0000 UTC m=+36.709608227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:17.975140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:17.975109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:18.084495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.084410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:18.450307 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.450272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:18.450307 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.450299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:18.450823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.450284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:18.456028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.456009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 11:15:18.457607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.457585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9kdff\""
Sep 10 11:15:18.457706 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.457638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 11:15:18.457760 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.457591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 11:15:18.457808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.457585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 11:15:18.458283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.458191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7sc2\""
Sep 10 11:15:18.631560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.631520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" event={"ID":"18362d46-4341-4ae6-b491-19876425251e","Type":"ContainerStarted","Data":"16216e0057f289ae5a24b7d75753d5a638b00a9aa33986c495e1ec9cf37c8b76"}
Sep 10 11:15:18.641560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.641534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:18.641707 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:18.641669    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:18.641758 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:18.641725    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:20.641706393 +0000 UTC m=+38.717032037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:18.814915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.814650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:18.944990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:18.944957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:20.638037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:20.637998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tq59" event={"ID":"c493c32a-781d-4ced-9053-f22d34da37af","Type":"ContainerStarted","Data":"498d6a61b36dba3f8cb922b491565f296fa2c383578e78de2104a4c187de562a"}
Sep 10 11:15:20.655686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:20.655653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:20.655817 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:20.655766    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:20.655855 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:20.655818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:24.655805552 +0000 UTC m=+42.731131199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:20.675037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:20.674992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kbbrz" podStartSLOduration=10.003047961 podStartE2EDuration="38.674972338s" totalImagesPullingTime="28.671924377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:18.666161129 +0000 UTC m=+36.741486793" watchObservedRunningTime="2026-09-10 11:15:20.674972338 +0000 UTC m=+38.750298003"
Sep 10 11:15:21.243983 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:21.243942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:21.336159 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:21.336133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:24.682736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:24.682697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:24.683237 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:24.682820    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:24.683237 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:24.682871    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:32.682857459 +0000 UTC m=+50.758183101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:25.151268 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:25.151232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:25.247304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:25.247270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:27.271747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.271694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6tq59" podStartSLOduration=8.680743127 podStartE2EDuration="11.271678933s" totalImagesPullingTime="2.590935806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:20.686834227 +0000 UTC m=+38.762159891" watchObservedRunningTime="2026-09-10 11:15:27.271678933 +0000 UTC m=+45.347004647"
Sep 10 11:15:27.272227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.271809    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8j76t"]
Sep 10 11:15:27.313490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.313458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8j76t"]
Sep 10 11:15:27.313643 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.313546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.319755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.319299    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 11:15:27.319755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.319537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 11:15:27.320516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.320501    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpt46\""
Sep 10 11:15:27.320588 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.320543    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 11:15:27.325010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.324984    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 11:15:27.327777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.327759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 11:15:27.404544 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-snapshots\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.404544 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-tmp\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.404728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjgl\" (UniqueName: \"kubernetes.io/projected/2dc89937-563f-4148-a495-37312d5beea1-kube-api-access-2xjgl\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.404728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.404728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.404728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.404713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89937-563f-4148-a495-37312d5beea1-serving-cert\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505177 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-tmp\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjgl\" (UniqueName: \"kubernetes.io/projected/2dc89937-563f-4148-a495-37312d5beea1-kube-api-access-2xjgl\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89937-563f-4148-a495-37312d5beea1-serving-cert\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505628 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-snapshots\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-tmp\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.505895 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.505876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.506050 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.506031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2dc89937-563f-4148-a495-37312d5beea1-snapshots\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.506135 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.506106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc89937-563f-4148-a495-37312d5beea1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.509284 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.509264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89937-563f-4148-a495-37312d5beea1-serving-cert\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.516935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.516914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjgl\" (UniqueName: \"kubernetes.io/projected/2dc89937-563f-4148-a495-37312d5beea1-kube-api-access-2xjgl\") pod \"insights-operator-76bcd6ccf5-8j76t\" (UID: \"2dc89937-563f-4148-a495-37312d5beea1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.571259 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.571173    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-glrhj"]
Sep 10 11:15:27.595298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.595260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-glrhj"]
Sep 10 11:15:27.595442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.595375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.599476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.599453    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xpwqk\""
Sep 10 11:15:27.599697 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.599678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 11:15:27.600247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.600233    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 11:15:27.600314 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.600295    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:15:27.605966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.605945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflrb\" (UniqueName: \"kubernetes.io/projected/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-kube-api-access-nflrb\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.606054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.605976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-config\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.606054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.606011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-serving-cert\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.606160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.606103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-trusted-ca\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.606210 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.606195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 11:15:27.612730 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.612716    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 11:15:27.621755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.621739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t"
Sep 10 11:15:27.683132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.681396    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"]
Sep 10 11:15:27.701520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.701485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"]
Sep 10 11:15:27.701648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.701624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.706722 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflrb\" (UniqueName: \"kubernetes.io/projected/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-kube-api-access-nflrb\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.706847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbqv\" (UniqueName: \"kubernetes.io/projected/ad76d830-b615-42cd-8786-e36046e4a7bc-kube-api-access-pjbqv\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.706847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-config\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.706964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-serving-cert\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.706964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.707067 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.706965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-trusted-ca\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.708871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.708128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-trusted-ca\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.708871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.708689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-config\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.712936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.711003    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 11:15:27.712936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.711303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 11:15:27.712936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.711324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-serving-cert\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.714386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.714164    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x6mjz\""
Sep 10 11:15:27.714386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.714212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:15:27.721535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.721474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"]
Sep 10 11:15:27.721535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.721509    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"]
Sep 10 11:15:27.721667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.721577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.725473 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.725405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflrb\" (UniqueName: \"kubernetes.io/projected/f59e35f6-4a05-4a79-86fd-f5aeaf409d61-kube-api-access-nflrb\") pod \"console-operator-85b454874b-glrhj\" (UID: \"f59e35f6-4a05-4a79-86fd-f5aeaf409d61\") " pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.725574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.725475    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 11:15:27.725723 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.725706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 11:15:27.725791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.725733    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gnx7f\""
Sep 10 11:15:27.725971 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.725955    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 11:15:27.728585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.728568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 11:15:27.807812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.807775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.807982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.807837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.807982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.807870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66rw\" (UniqueName: \"kubernetes.io/projected/9e478389-5b0b-44e8-a48a-3eb07b761b8b-kube-api-access-b66rw\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.807982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.807890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbqv\" (UniqueName: \"kubernetes.io/projected/ad76d830-b615-42cd-8786-e36046e4a7bc-kube-api-access-pjbqv\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.807982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.807906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9e478389-5b0b-44e8-a48a-3eb07b761b8b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.807982 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:27.807917    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:27.808178 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:27.807993    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:28.307976051 +0000 UTC m=+46.383301695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:27.835567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.835491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbqv\" (UniqueName: \"kubernetes.io/projected/ad76d830-b615-42cd-8786-e36046e4a7bc-kube-api-access-pjbqv\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:27.904486 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.904456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:27.908869 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.908840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.908989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.908879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66rw\" (UniqueName: \"kubernetes.io/projected/9e478389-5b0b-44e8-a48a-3eb07b761b8b-kube-api-access-b66rw\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.908989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.908907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9e478389-5b0b-44e8-a48a-3eb07b761b8b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.909125 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:27.908998    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:27.909125 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:27.909063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:28.409046737 +0000 UTC m=+46.484372381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:27.909661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.909637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9e478389-5b0b-44e8-a48a-3eb07b761b8b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.939860 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.939841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66rw\" (UniqueName: \"kubernetes.io/projected/9e478389-5b0b-44e8-a48a-3eb07b761b8b-kube-api-access-b66rw\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:27.948117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:27.948075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8j76t"]
Sep 10 11:15:27.951115 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:27.951068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc89937_563f_4148_a495_37312d5beea1.slice/crio-59774230686a2eff557efde02ef14d74aab384dfe010236cafa8542f907a7146 WatchSource:0}: Error finding container 59774230686a2eff557efde02ef14d74aab384dfe010236cafa8542f907a7146: Status 404 returned error can't find the container with id 59774230686a2eff557efde02ef14d74aab384dfe010236cafa8542f907a7146
Sep 10 11:15:28.184133 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.184102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:28.258718 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.258677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-glrhj"]
Sep 10 11:15:28.261861 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:28.261822    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59e35f6_4a05_4a79_86fd_f5aeaf409d61.slice/crio-ea4ae109bbe006c628a32020f9c64440f1efc09976116ecc48780771a2c26cc8 WatchSource:0}: Error finding container ea4ae109bbe006c628a32020f9c64440f1efc09976116ecc48780771a2c26cc8: Status 404 returned error can't find the container with id ea4ae109bbe006c628a32020f9c64440f1efc09976116ecc48780771a2c26cc8
Sep 10 11:15:28.312918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.312885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:28.313303 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:28.313031    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:28.313303 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:28.313123    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:29.313108841 +0000 UTC m=+47.388434497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:28.378782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.378747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:28.413911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.413876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:28.414130 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:28.414056    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:28.414206 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:28.414160    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:29.414140086 +0000 UTC m=+47.489465730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:28.657197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.657166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-glrhj" event={"ID":"f59e35f6-4a05-4a79-86fd-f5aeaf409d61","Type":"ContainerStarted","Data":"ea4ae109bbe006c628a32020f9c64440f1efc09976116ecc48780771a2c26cc8"}
Sep 10 11:15:28.658516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:28.658489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t" event={"ID":"2dc89937-563f-4148-a495-37312d5beea1","Type":"ContainerStarted","Data":"59774230686a2eff557efde02ef14d74aab384dfe010236cafa8542f907a7146"}
Sep 10 11:15:29.321390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:29.321351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:29.321805 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:29.321520    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:29.321805 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:29.321595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:31.321574599 +0000 UTC m=+49.396900245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:29.421794 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:29.421757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:29.421973 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:29.421955    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:29.422041 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:29.422036    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:31.422015991 +0000 UTC m=+49.497341648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:29.951046 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:29.951005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:30.068963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:30.068932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:31.339367 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.339311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:31.339654 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:31.339471    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:31.339654 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:31.339543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:35.339522888 +0000 UTC m=+53.414848548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:31.440469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.440425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:31.440664 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:31.440554    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:31.440664 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:31.440618    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:35.440597238 +0000 UTC m=+53.515922895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:31.668913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.668880    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/0.log"
Sep 10 11:15:31.669109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.668929    2578 generic.go:356] "Generic (PLEG): container finished" podID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61" containerID="cfa664d85e4bbdfa37d52f338b4f850c896693be99d8f4ea0ee1c03cd741ed70" exitCode=255
Sep 10 11:15:31.669109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.669005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-glrhj" event={"ID":"f59e35f6-4a05-4a79-86fd-f5aeaf409d61","Type":"ContainerDied","Data":"cfa664d85e4bbdfa37d52f338b4f850c896693be99d8f4ea0ee1c03cd741ed70"}
Sep 10 11:15:31.669336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.669310    2578 scope.go:117] "RemoveContainer" containerID="cfa664d85e4bbdfa37d52f338b4f850c896693be99d8f4ea0ee1c03cd741ed70"
Sep 10 11:15:31.670819 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.670794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t" event={"ID":"2dc89937-563f-4148-a495-37312d5beea1","Type":"ContainerStarted","Data":"af4a7d1fce2910a3b80eb5efa72ab120f754b1ee1ecb09004d84d1b753820925"}
Sep 10 11:15:31.719617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:31.719574    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t" podStartSLOduration=1.725585634 podStartE2EDuration="4.719555227s" totalImagesPullingTime="2.993969593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:31.718415814 +0000 UTC m=+49.793741478" watchObservedRunningTime="2026-09-10 11:15:31.719555227 +0000 UTC m=+49.794880894"
Sep 10 11:15:32.013374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.013289    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-c8rl2"]
Sep 10 11:15:32.023008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.022981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2"
Sep 10 11:15:32.027679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.027653    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5m5x6\""
Sep 10 11:15:32.033048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.033014    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-c8rl2"]
Sep 10 11:15:32.046012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.045981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pcx\" (UniqueName: \"kubernetes.io/projected/f30ec824-e226-4b7f-bda8-c6b31dc4ddf2-kube-api-access-q9pcx\") pod \"network-check-source-685456655f-c8rl2\" (UID: \"f30ec824-e226-4b7f-bda8-c6b31dc4ddf2\") " pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2"
Sep 10 11:15:32.050857 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.050817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:32.094544 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.094504    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h5wlr"]
Sep 10 11:15:32.110534 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.110502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.115121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.114681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tf5wn\""
Sep 10 11:15:32.115121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.114845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 11:15:32.115121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.114989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 11:15:32.116240 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.116215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h5wlr"]
Sep 10 11:15:32.147284 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5ec7ff7-d754-48ce-a296-be92da83561f-crio-socket\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.147439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.147439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pcx\" (UniqueName: \"kubernetes.io/projected/f30ec824-e226-4b7f-bda8-c6b31dc4ddf2-kube-api-access-q9pcx\") pod \"network-check-source-685456655f-c8rl2\" (UID: \"f30ec824-e226-4b7f-bda8-c6b31dc4ddf2\") " pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2"
Sep 10 11:15:32.147439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5ec7ff7-d754-48ce-a296-be92da83561f-data-volume\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.147540 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtzr\" (UniqueName: \"kubernetes.io/projected/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-api-access-qrtzr\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.147540 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.147475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.159886 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.159837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pcx\" (UniqueName: \"kubernetes.io/projected/f30ec824-e226-4b7f-bda8-c6b31dc4ddf2-kube-api-access-q9pcx\") pod \"network-check-source-685456655f-c8rl2\" (UID: \"f30ec824-e226-4b7f-bda8-c6b31dc4ddf2\") " pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2"
Sep 10 11:15:32.169744 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.169685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:32.248779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.248745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5ec7ff7-d754-48ce-a296-be92da83561f-crio-socket\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.248954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.248795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.248954 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.248890    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:15:32.248954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.248948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5ec7ff7-d754-48ce-a296-be92da83561f-data-volume\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.249138 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.248963    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:32.74894555 +0000 UTC m=+50.824271207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : configmap "kube-rbac-proxy" not found
Sep 10 11:15:32.249138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.248987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtzr\" (UniqueName: \"kubernetes.io/projected/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-api-access-qrtzr\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.249138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.249009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5ec7ff7-d754-48ce-a296-be92da83561f-crio-socket\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.249138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.249021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.249138 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.249117    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:32.249326 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.249167    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:32.749154403 +0000 UTC m=+50.824480046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:32.249364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.249325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5ec7ff7-d754-48ce-a296-be92da83561f-data-volume\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.264176 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.264114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtzr\" (UniqueName: \"kubernetes.io/projected/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-api-access-qrtzr\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.330928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.330888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2"
Sep 10 11:15:32.625368 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:32.625341    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30ec824_e226_4b7f_bda8_c6b31dc4ddf2.slice/crio-fd6b9ed91e88c296061e54aa87dd8785a46c2e8be8a6e0c770cec31339219bbd WatchSource:0}: Error finding container fd6b9ed91e88c296061e54aa87dd8785a46c2e8be8a6e0c770cec31339219bbd: Status 404 returned error can't find the container with id fd6b9ed91e88c296061e54aa87dd8785a46c2e8be8a6e0c770cec31339219bbd
Sep 10 11:15:32.625762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.625447    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-c8rl2"]
Sep 10 11:15:32.675126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675079    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/1.log"
Sep 10 11:15:32.675528 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675512    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/0.log"
Sep 10 11:15:32.675581 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675546    2578 generic.go:356] "Generic (PLEG): container finished" podID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48" exitCode=255
Sep 10 11:15:32.675633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-glrhj" event={"ID":"f59e35f6-4a05-4a79-86fd-f5aeaf409d61","Type":"ContainerDied","Data":"3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"}
Sep 10 11:15:32.675686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675671    2578 scope.go:117] "RemoveContainer" containerID="cfa664d85e4bbdfa37d52f338b4f850c896693be99d8f4ea0ee1c03cd741ed70"
Sep 10 11:15:32.675889 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.675874    2578 scope.go:117] "RemoveContainer" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"
Sep 10 11:15:32.676063 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.676036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:15:32.676898 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.676878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2" event={"ID":"f30ec824-e226-4b7f-bda8-c6b31dc4ddf2","Type":"ContainerStarted","Data":"fd6b9ed91e88c296061e54aa87dd8785a46c2e8be8a6e0c770cec31339219bbd"}
Sep 10 11:15:32.753340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.753305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.753488 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753390    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:15:32.753488 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753448    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:33.753422055 +0000 UTC m=+51.828747702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : configmap "kube-rbac-proxy" not found
Sep 10 11:15:32.753624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.753599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:32.753681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:32.753669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:32.753729 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753680    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:32.753783 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753751    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:15:48.753732125 +0000 UTC m=+66.829057777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:32.753849 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753783    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:32.753903 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:32.753853    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:33.753837566 +0000 UTC m=+51.829163242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:33.682124 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:33.682073    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/1.log"
Sep 10 11:15:33.682552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:33.682460    2578 scope.go:117] "RemoveContainer" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"
Sep 10 11:15:33.682645 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:33.682627    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:15:33.747574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:33.747547    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bbh66_8b13de39-1c19-4465-9e6c-2d077b260bd5/dns-node-resolver/0.log"
Sep 10 11:15:33.761935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:33.761905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:33.762122 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:33.762076    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:33.762197 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:33.762165    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:35.762146048 +0000 UTC m=+53.837471696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:33.762268 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:33.762218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:33.762322 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:33.762315    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:15:33.762374 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:33.762356    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:35.762343714 +0000 UTC m=+53.837669379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : configmap "kube-rbac-proxy" not found
Sep 10 11:15:34.689867 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:34.689828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:34.748250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:34.748218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm6c6_85c3fa39-d36c-470b-99f0-14820da3b3dd/node-ca/0.log"
Sep 10 11:15:34.780218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:34.780178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:34.910156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:34.910108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:35.375560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.375520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:35.375758 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.375710    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:35.375818 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.375787    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:43.375771257 +0000 UTC m=+61.451096923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:35.476583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.476543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:35.476791 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.476703    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:35.476791 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.476786    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:43.476767371 +0000 UTC m=+61.552093035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:35.689422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.689381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2" event={"ID":"f30ec824-e226-4b7f-bda8-c6b31dc4ddf2","Type":"ContainerStarted","Data":"97de75ee27a6f793c02624cdd816054bc3bbc4a2333d60ba24fccde5c6c6dd82"}
Sep 10 11:15:35.715707 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.715655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-c8rl2" podStartSLOduration=1.7220297759999998 podStartE2EDuration="4.71549415s" totalImagesPullingTime="2.993464374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:35.713974292 +0000 UTC m=+53.789299958" watchObservedRunningTime="2026-09-10 11:15:35.71549415 +0000 UTC m=+53.790819815"
Sep 10 11:15:35.748558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.748527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tq59_c493c32a-781d-4ced-9053-f22d34da37af/serve-healthcheck-canary/0.log"
Sep 10 11:15:35.779166 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.779127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:35.779330 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:35.779222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:35.779330 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.779299    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:35.779330 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.779324    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:15:35.779467 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.779369    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:39.779350999 +0000 UTC m=+57.854676644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:35.779467 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:35.779386    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:39.779378213 +0000 UTC m=+57.854703854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : configmap "kube-rbac-proxy" not found
Sep 10 11:15:37.707247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.707210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:37.817218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.817188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:37.905642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.905597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:37.905806 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.905692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:37.906015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.906004    2578 scope.go:117] "RemoveContainer" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"
Sep 10 11:15:37.906190 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:37.906171    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:15:37.975836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:37.975756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:38.295049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:38.294960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:38.395309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:38.395276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:38.681056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:38.681028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:38.799954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:38.799916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:39.188163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:39.188129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:39.283874 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:39.283846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:39.811146 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:39.811079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:39.811537 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:39.811199    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:39.811537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:39.811226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:39.811537 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:39.811263    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:47.811242795 +0000 UTC m=+65.886568444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:39.811537 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:39.811278    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:15:39.811537 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:39.811328    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:15:47.811315878 +0000 UTC m=+65.886641519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : configmap "kube-rbac-proxy" not found
Sep 10 11:15:40.329999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:40.329965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:40.461788 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:40.461754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:41.567226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:41.567195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:41.612757 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:41.612731    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c24rb"
Sep 10 11:15:43.442434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:43.442398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:43.442851 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:43.442512    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 11:15:43.442851 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:43.442567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls podName:ad76d830-b615-42cd-8786-e36046e4a7bc nodeName:}" failed. No retries permitted until 2026-09-10 11:15:59.44255295 +0000 UTC m=+77.517878592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-l4x5l" (UID: "ad76d830-b615-42cd-8786-e36046e4a7bc") : secret "samples-operator-tls" not found
Sep 10 11:15:43.543079 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:43.543039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:43.543262 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:43.543195    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:43.543304 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:43.543261    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls podName:9e478389-5b0b-44e8-a48a-3eb07b761b8b nodeName:}" failed. No retries permitted until 2026-09-10 11:15:59.543245536 +0000 UTC m=+77.618571201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6drlq" (UID: "9e478389-5b0b-44e8-a48a-3eb07b761b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 10 11:15:45.160030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:45.159994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:45.296746 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:45.296718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:47.878258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:47.878216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:47.878761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:47.878298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:47.878761 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:47.878392    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:47.878761 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:47.878479    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls podName:c5ec7ff7-d754-48ce-a296-be92da83561f nodeName:}" failed. No retries permitted until 2026-09-10 11:16:03.878457045 +0000 UTC m=+81.953782690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5wlr" (UID: "c5ec7ff7-d754-48ce-a296-be92da83561f") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:15:47.878926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:47.878765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5ec7ff7-d754-48ce-a296-be92da83561f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:15:48.180525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.180437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:15:48.183867 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.183847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 11:15:48.191626 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:48.191609    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 11:15:48.191695 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:48.191660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs podName:8b95e13d-b128-4c10-b009-e54830d2be14 nodeName:}" failed. No retries permitted until 2026-09-10 11:16:52.191644995 +0000 UTC m=+130.266970641 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs") pod "network-metrics-daemon-xmwgb" (UID: "8b95e13d-b128-4c10-b009-e54830d2be14") : secret "metrics-daemon-secret" not found
Sep 10 11:15:48.280973 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.280936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:48.284657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.284639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 11:15:48.293831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.293803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34887360-7db4-44aa-8c7a-976d77303b5b-original-pull-secret\") pod \"global-pull-secret-syncer-dh54m\" (UID: \"34887360-7db4-44aa-8c7a-976d77303b5b\") " pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:48.382176 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.382121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:48.384715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.384696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnpr\" (UniqueName: \"kubernetes.io/projected/9e0341b5-bfe5-4b77-a855-9fd30af1817f-kube-api-access-mvnpr\") pod \"network-check-target-zczmj\" (UID: \"9e0341b5-bfe5-4b77-a855-9fd30af1817f\") " pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:48.472308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.472236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7sc2\""
Sep 10 11:15:48.474233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.474217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh54m"
Sep 10 11:15:48.479479 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.479459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:48.759538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.759516    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh54m"]
Sep 10 11:15:48.761262 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:48.761234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34887360_7db4_44aa_8c7a_976d77303b5b.slice/crio-67a094116b625909de8d5b734a62fe7d689c95818e2979b565aad50cc43159ef WatchSource:0}: Error finding container 67a094116b625909de8d5b734a62fe7d689c95818e2979b565aad50cc43159ef: Status 404 returned error can't find the container with id 67a094116b625909de8d5b734a62fe7d689c95818e2979b565aad50cc43159ef
Sep 10 11:15:48.770976 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:15:48.770950    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0341b5_bfe5_4b77_a855_9fd30af1817f.slice/crio-d17355110c580d26d85760adc0a8a1226087fd1f38ee75c85eee15694658bd60 WatchSource:0}: Error finding container d17355110c580d26d85760adc0a8a1226087fd1f38ee75c85eee15694658bd60: Status 404 returned error can't find the container with id d17355110c580d26d85760adc0a8a1226087fd1f38ee75c85eee15694658bd60
Sep 10 11:15:48.771453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.771434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zczmj"]
Sep 10 11:15:48.785474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:48.785449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:15:48.785616 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:48.785599    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:15:48.785683 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:48.785666    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls podName:5e3b4414-485e-4bf7-8743-c0b0b6a47226 nodeName:}" failed. No retries permitted until 2026-09-10 11:16:20.785645899 +0000 UTC m=+98.860971557 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls") pod "dns-default-dbfxf" (UID: "5e3b4414-485e-4bf7-8743-c0b0b6a47226") : secret "dns-default-metrics-tls" not found
Sep 10 11:15:49.725196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:49.725163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zczmj" event={"ID":"9e0341b5-bfe5-4b77-a855-9fd30af1817f","Type":"ContainerStarted","Data":"7af98bdfa45f17f55ca2e5c81f5080fd7a789c57ca1733cd407af8bcfdf119ad"}
Sep 10 11:15:49.725642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:49.725303    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:15:49.725642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:49.725328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zczmj" event={"ID":"9e0341b5-bfe5-4b77-a855-9fd30af1817f","Type":"ContainerStarted","Data":"d17355110c580d26d85760adc0a8a1226087fd1f38ee75c85eee15694658bd60"}
Sep 10 11:15:49.726301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:49.726279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh54m" event={"ID":"34887360-7db4-44aa-8c7a-976d77303b5b","Type":"ContainerStarted","Data":"67a094116b625909de8d5b734a62fe7d689c95818e2979b565aad50cc43159ef"}
Sep 10 11:15:49.755596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:49.755551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zczmj" podStartSLOduration=67.755537428 podStartE2EDuration="1m7.755537428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:49.755220611 +0000 UTC m=+67.830546286" watchObservedRunningTime="2026-09-10 11:15:49.755537428 +0000 UTC m=+67.830863133"
Sep 10 11:15:51.142039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:51.142004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:52.112243 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:52.112210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:52.202126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:52.202069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:52.308002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:52.307959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:52.451761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:52.451686    2578 scope.go:117] "RemoveContainer" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"
Sep 10 11:15:53.180932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.180892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:53.361409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.361376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:53.717673 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.717645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:53.741436 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.741358    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:15:53.742167 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.742077    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/1.log"
Sep 10 11:15:53.742278 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.742186    2578 generic.go:356] "Generic (PLEG): container finished" podID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61" containerID="1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488" exitCode=255
Sep 10 11:15:53.742278 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.742262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-glrhj" event={"ID":"f59e35f6-4a05-4a79-86fd-f5aeaf409d61","Type":"ContainerDied","Data":"1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488"}
Sep 10 11:15:53.742375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.742321    2578 scope.go:117] "RemoveContainer" containerID="3bce8c0133b5c3747e452695046ad16f6785fcbfbf636954c6b06b9ce30cda48"
Sep 10 11:15:53.742704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.742684    2578 scope.go:117] "RemoveContainer" containerID="1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488"
Sep 10 11:15:53.742905 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:53.742881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:15:53.743962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.743934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh54m" event={"ID":"34887360-7db4-44aa-8c7a-976d77303b5b","Type":"ContainerStarted","Data":"c434c8684be7aa263edc4861edba2de6e40605bede1a89b51dee96d18d6e9837"}
Sep 10 11:15:53.785445 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.785401    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dh54m" podStartSLOduration=66.994974037 podStartE2EDuration="1m11.785386819s" totalImagesPullingTime="4.790412782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:15:53.784050484 +0000 UTC m=+71.859376147" watchObservedRunningTime="2026-09-10 11:15:53.785386819 +0000 UTC m=+71.860712483"
Sep 10 11:15:53.913815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:53.913780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:54.011888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:54.011821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:54.251030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:54.251002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:54.555740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:54.555707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:54.748791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:54.748767    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:15:54.923681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:54.923654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:55.106575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:55.106547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:55.509687 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:55.509653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:56.101574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:56.101551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:56.312506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:56.312476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:56.524406 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:56.524373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:56.911097 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:56.911052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:57.164505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:57.164423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:57.510215 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:57.510140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:57.905417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:57.905386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:57.905553 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:57.905437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:15:57.905734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:57.905722    2578 scope.go:117] "RemoveContainer" containerID="1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488"
Sep 10 11:15:57.905900 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:15:57.905881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:15:58.127541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:58.127508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:58.505636 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:58.505601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:58.709449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:58.709418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:59.135861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.135831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:15:59.469622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.469513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:59.472178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.472156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad76d830-b615-42cd-8786-e36046e4a7bc-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-l4x5l\" (UID: \"ad76d830-b615-42cd-8786-e36046e4a7bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:59.522777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.522735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x6mjz\""
Sep 10 11:15:59.530527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.530495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"
Sep 10 11:15:59.570390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.570355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:59.572926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.572905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e478389-5b0b-44e8-a48a-3eb07b761b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6drlq\" (UID: \"9e478389-5b0b-44e8-a48a-3eb07b761b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:59.808995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.808907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l"]
Sep 10 11:15:59.834246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.834216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gnx7f\""
Sep 10 11:15:59.840892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.840866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"
Sep 10 11:15:59.903349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:15:59.903319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:00.127851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.127820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:00.132416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.132391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq"]
Sep 10 11:16:00.135388 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:00.135365    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e478389_5b0b_44e8_a48a_3eb07b761b8b.slice/crio-eccea903061163295f0e26c3287899a57ee40279fe48aa85eea80445db7bcb79 WatchSource:0}: Error finding container eccea903061163295f0e26c3287899a57ee40279fe48aa85eea80445db7bcb79: Status 404 returned error can't find the container with id eccea903061163295f0e26c3287899a57ee40279fe48aa85eea80445db7bcb79
Sep 10 11:16:00.230725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.230693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:00.335923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.335890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:00.526492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.526409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:00.769187 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.769153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l" event={"ID":"ad76d830-b615-42cd-8786-e36046e4a7bc","Type":"ContainerStarted","Data":"3a678b3c2f00983edff2d87a88bb8269bd0720b02f8dca7d1ad225dbbb5176b2"}
Sep 10 11:16:00.770544 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.770514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq" event={"ID":"9e478389-5b0b-44e8-a48a-3eb07b761b8b","Type":"ContainerStarted","Data":"eccea903061163295f0e26c3287899a57ee40279fe48aa85eea80445db7bcb79"}
Sep 10 11:16:00.972822 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:00.972754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:01.228559 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.227533    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"]
Sep 10 11:16:01.233861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.233834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.242196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.242076    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rtzz2"]
Sep 10 11:16:01.243975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.243954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 11:16:01.244557 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.244537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 11:16:01.244652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.244568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wrsj4\""
Sep 10 11:16:01.245292 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.245271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.246169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.245751    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 11:16:01.246169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.245986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 11:16:01.248100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.248064    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 11:16:01.250524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.250507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w6jrn\""
Sep 10 11:16:01.250767 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.250744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 11:16:01.274564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.274521    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"]
Sep 10 11:16:01.278212 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.278184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"]
Sep 10 11:16:01.278358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.278221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rtzz2"]
Sep 10 11:16:01.278358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.278326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.287861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.287840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 11:16:01.288105 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.288072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 11:16:01.288265 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.288243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 11:16:01.342210 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.342174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 11:16:01.367913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.367882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"]
Sep 10 11:16:01.388558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.388721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f13d5f-d575-4d67-9ee3-9edb48355209-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.388721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-nginx-conf\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.388721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.388848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.388848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.388951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5w4\" (UniqueName: \"kubernetes.io/projected/6afeb847-f02b-4cab-8887-da080da7f87c-kube-api-access-wb5w4\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.388951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afeb847-f02b-4cab-8887-da080da7f87c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.389059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.388980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnt4f\" (UniqueName: \"kubernetes.io/projected/89f13d5f-d575-4d67-9ee3-9edb48355209-kube-api-access-gnt4f\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.389059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.389013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.489876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.489876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f13d5f-d575-4d67-9ee3-9edb48355209-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.489876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-nginx-conf\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.489876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.489977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5w4\" (UniqueName: \"kubernetes.io/projected/6afeb847-f02b-4cab-8887-da080da7f87c-kube-api-access-wb5w4\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.490000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afeb847-f02b-4cab-8887-da080da7f87c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.490051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnt4f\" (UniqueName: \"kubernetes.io/projected/89f13d5f-d575-4d67-9ee3-9edb48355209-kube-api-access-gnt4f\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.490077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.491684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.491652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afeb847-f02b-4cab-8887-da080da7f87c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.491802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.491748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-nginx-conf\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.493381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.493346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.493622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.493600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-ca\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.493721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.493667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-hub\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.493982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.493961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6afeb847-f02b-4cab-8887-da080da7f87c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.494146 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.494122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89f13d5f-d575-4d67-9ee3-9edb48355209-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.494278 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.494254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e175bf87-32d0-422a-b5ea-4dd6a5b58fb5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rtzz2\" (UID: \"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.560045 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.560013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2"
Sep 10 11:16:01.561869 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.561849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5w4\" (UniqueName: \"kubernetes.io/projected/6afeb847-f02b-4cab-8887-da080da7f87c-kube-api-access-wb5w4\") pod \"cluster-proxy-proxy-agent-5798bc5b4d-mztrs\" (UID: \"6afeb847-f02b-4cab-8887-da080da7f87c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.563213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.563192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:01.569992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.569968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnt4f\" (UniqueName: \"kubernetes.io/projected/89f13d5f-d575-4d67-9ee3-9edb48355209-kube-api-access-gnt4f\") pod \"managed-serviceaccount-addon-agent-6ccd96b48b-qrccc\" (UID: \"89f13d5f-d575-4d67-9ee3-9edb48355209\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.590717 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.590681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"
Sep 10 11:16:01.852037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.851955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"
Sep 10 11:16:01.881231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:01.881195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:02.398168 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:02.398117    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode175bf87_32d0_422a_b5ea_4dd6a5b58fb5.slice/crio-c24cb2a36ec20d900f0d863f60114666363e34eea6c685450bd70e8737fd5e57 WatchSource:0}: Error finding container c24cb2a36ec20d900f0d863f60114666363e34eea6c685450bd70e8737fd5e57: Status 404 returned error can't find the container with id c24cb2a36ec20d900f0d863f60114666363e34eea6c685450bd70e8737fd5e57
Sep 10 11:16:02.398615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.398508    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rtzz2"]
Sep 10 11:16:02.398615 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:02.398597    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f13d5f_d575_4d67_9ee3_9edb48355209.slice/crio-c46e191004dc16745cb711cc1cac593f20cf2c9578acdd3377908698ee9af35f WatchSource:0}: Error finding container c46e191004dc16745cb711cc1cac593f20cf2c9578acdd3377908698ee9af35f: Status 404 returned error can't find the container with id c46e191004dc16745cb711cc1cac593f20cf2c9578acdd3377908698ee9af35f
Sep 10 11:16:02.399598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.399576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc"]
Sep 10 11:16:02.410298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.410276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs"]
Sep 10 11:16:02.410738 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:02.410720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6afeb847_f02b_4cab_8887_da080da7f87c.slice/crio-404a1af5601ce52a5bbbb0e235338784115b3ee7d92fb6b7db937a3a2435c98a WatchSource:0}: Error finding container 404a1af5601ce52a5bbbb0e235338784115b3ee7d92fb6b7db937a3a2435c98a: Status 404 returned error can't find the container with id 404a1af5601ce52a5bbbb0e235338784115b3ee7d92fb6b7db937a3a2435c98a
Sep 10 11:16:02.649413 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.649320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:02.778144 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.778104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs" event={"ID":"6afeb847-f02b-4cab-8887-da080da7f87c","Type":"ContainerStarted","Data":"404a1af5601ce52a5bbbb0e235338784115b3ee7d92fb6b7db937a3a2435c98a"}
Sep 10 11:16:02.779329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.779299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc" event={"ID":"89f13d5f-d575-4d67-9ee3-9edb48355209","Type":"ContainerStarted","Data":"c46e191004dc16745cb711cc1cac593f20cf2c9578acdd3377908698ee9af35f"}
Sep 10 11:16:02.781117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.781076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l" event={"ID":"ad76d830-b615-42cd-8786-e36046e4a7bc","Type":"ContainerStarted","Data":"5acc2235ba8b04646913d7ecd133a969944b3944e5d97977369ebd650eda8933"}
Sep 10 11:16:02.781198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.781142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l" event={"ID":"ad76d830-b615-42cd-8786-e36046e4a7bc","Type":"ContainerStarted","Data":"f7f0b32783768c8eb9ffeb4b00dddbb493668b52f8bfe0bb29eb36f2cb30b0f3"}
Sep 10 11:16:02.782556 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.782536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq" event={"ID":"9e478389-5b0b-44e8-a48a-3eb07b761b8b","Type":"ContainerStarted","Data":"32fbb7bee31f1170ee3ae4b983e7d979ac0502b0e3d07f4d5abec2cb8c3b9678"}
Sep 10 11:16:02.783720 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.783702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2" event={"ID":"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5","Type":"ContainerStarted","Data":"c24cb2a36ec20d900f0d863f60114666363e34eea6c685450bd70e8737fd5e57"}
Sep 10 11:16:02.926659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.926553    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-l4x5l" podStartSLOduration=33.688508828 podStartE2EDuration="35.926534676s" totalImagesPullingTime="2.238025848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:02.827372199 +0000 UTC m=+80.902697862" watchObservedRunningTime="2026-09-10 11:16:02.926534676 +0000 UTC m=+81.001860347"
Sep 10 11:16:02.926659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:02.926636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6drlq" podStartSLOduration=33.96416439 podStartE2EDuration="35.926631368s" totalImagesPullingTime="1.962466978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:02.916417147 +0000 UTC m=+80.991742809" watchObservedRunningTime="2026-09-10 11:16:02.926631368 +0000 UTC m=+81.001957114"
Sep 10 11:16:03.111337 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.111307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:03.351422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.351264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:03.916658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.912050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:16:03.916658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.915861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5ec7ff7-d754-48ce-a296-be92da83561f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5wlr\" (UID: \"c5ec7ff7-d754-48ce-a296-be92da83561f\") " pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:16:03.923928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.923901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tf5wn\""
Sep 10 11:16:03.931281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:03.931095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h5wlr"
Sep 10 11:16:04.240052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:04.240026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h5wlr"]
Sep 10 11:16:04.242544 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:04.242519    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ec7ff7_d754_48ce_a296_be92da83561f.slice/crio-9b21f16711d2d3fe6aea7cccd5bf56a5e20e5c916484a521f644da41ca1f9b81 WatchSource:0}: Error finding container 9b21f16711d2d3fe6aea7cccd5bf56a5e20e5c916484a521f644da41ca1f9b81: Status 404 returned error can't find the container with id 9b21f16711d2d3fe6aea7cccd5bf56a5e20e5c916484a521f644da41ca1f9b81
Sep 10 11:16:04.799004 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:04.798966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5wlr" event={"ID":"c5ec7ff7-d754-48ce-a296-be92da83561f","Type":"ContainerStarted","Data":"e94eb53d2bcef2a1e0e969c325dbd6033f73e93e00ccdb70f51bd42bd160bb51"}
Sep 10 11:16:04.799247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:04.799034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5wlr" event={"ID":"c5ec7ff7-d754-48ce-a296-be92da83561f","Type":"ContainerStarted","Data":"9b21f16711d2d3fe6aea7cccd5bf56a5e20e5c916484a521f644da41ca1f9b81"}
Sep 10 11:16:04.800990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:04.800966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2" event={"ID":"e175bf87-32d0-422a-b5ea-4dd6a5b58fb5","Type":"ContainerStarted","Data":"be2a1e2604c442f6f2bf3d5fd54c576eb44aa7baecbcd559b2f8238430aa6865"}
Sep 10 11:16:05.020704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:05.020596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:05.150172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:05.150133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:07.812375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.812336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc" event={"ID":"89f13d5f-d575-4d67-9ee3-9edb48355209","Type":"ContainerStarted","Data":"db150444fa4a1bbdc0521104e1809a13a74847f8a2a02f16e63f477789288277"}
Sep 10 11:16:07.814171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.814149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5wlr" event={"ID":"c5ec7ff7-d754-48ce-a296-be92da83561f","Type":"ContainerStarted","Data":"2a9802ff8ac5ff46cb424527157d50ef59616ef246489e87dfcb5fc4af71bc6c"}
Sep 10 11:16:07.815919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.815893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs" event={"ID":"6afeb847-f02b-4cab-8887-da080da7f87c","Type":"ContainerStarted","Data":"2ef6318764fc1987bd91f528ad3e09bc19fa435a4c38b3b9a5739d3192ec6aaf"}
Sep 10 11:16:07.816008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.815927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs" event={"ID":"6afeb847-f02b-4cab-8887-da080da7f87c","Type":"ContainerStarted","Data":"5c994e032c1d038cca59cb9317ead4565fda30d07c7f88cd84800173c9ec498b"}
Sep 10 11:16:07.816008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.815940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs" event={"ID":"6afeb847-f02b-4cab-8887-da080da7f87c","Type":"ContainerStarted","Data":"840c63e65c74d1db7b15c38c34bed6fcab4b441d10e09d875339318513d3b2e9"}
Sep 10 11:16:07.840403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.840363    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-rtzz2" podStartSLOduration=5.390923315 podStartE2EDuration="6.840347454s" totalImagesPullingTime="1.449424139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:04.835642299 +0000 UTC m=+82.910967964" watchObservedRunningTime="2026-09-10 11:16:07.840347454 +0000 UTC m=+85.915673120"
Sep 10 11:16:07.840939 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:07.840905    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccd96b48b-qrccc" podStartSLOduration=2.032247988 podStartE2EDuration="6.840896782s" totalImagesPullingTime="4.808648794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:07.840245907 +0000 UTC m=+85.915571572" watchObservedRunningTime="2026-09-10 11:16:07.840896782 +0000 UTC m=+85.916222446"
Sep 10 11:16:08.450026 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:08.449994    2578 scope.go:117] "RemoveContainer" containerID="1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488"
Sep 10 11:16:08.450268 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:16:08.450240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-glrhj_openshift-console-operator(f59e35f6-4a05-4a79-86fd-f5aeaf409d61)\"" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podUID="f59e35f6-4a05-4a79-86fd-f5aeaf409d61"
Sep 10 11:16:09.826384 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:09.826301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5wlr" event={"ID":"c5ec7ff7-d754-48ce-a296-be92da83561f","Type":"ContainerStarted","Data":"07d3a21d5f9e00b18a14e679ccdc4be75107a680bc0c84e1cf5194fd7e6a7bb4"}
Sep 10 11:16:09.860324 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:09.858636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5798bc5b4d-mztrs" podStartSLOduration=4.027312044 podStartE2EDuration="8.858615263s" totalImagesPullingTime="4.831303219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:07.880284103 +0000 UTC m=+85.955609766" watchObservedRunningTime="2026-09-10 11:16:09.858615263 +0000 UTC m=+87.933940957"
Sep 10 11:16:12.954438 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:12.954405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:13.058663 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:13.058631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:13.197883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:13.197850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:13.311727 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:13.311635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:13.395568 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:13.395532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:15.268496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:15.268457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:17.439408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.439376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:17.665286 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.665245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:17.768658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.768549    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h5wlr" podStartSLOduration=40.670144105 podStartE2EDuration="45.768532467s" totalImagesPullingTime="5.098388362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:15:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:09.860847855 +0000 UTC m=+87.936173519" watchObservedRunningTime="2026-09-10 11:16:17.768532467 +0000 UTC m=+95.843858129"
Sep 10 11:16:17.769238 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.769216    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vtwh4"]
Sep 10 11:16:17.772684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.772662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.777994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.777968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 11:16:17.778930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.778908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 11:16:17.779454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.779223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 11:16:17.779454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.779305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cd9j5\""
Sep 10 11:16:17.779614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.779589    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 11:16:17.828717 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.828682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:17.926485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-wtmp\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-metrics-client-ca\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-textfile\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-root\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plhn\" (UniqueName: \"kubernetes.io/projected/6b671393-bc0d-4047-bdf7-2da219e22717-kube-api-access-5plhn\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-sys\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.926835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.926776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:17.931495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:17.931470    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.027763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.027763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-wtmp\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.027763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-metrics-client-ca\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-textfile\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-root\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:16:18.027859    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-root\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-wtmp\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:16:18.027926    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls podName:6b671393-bc0d-4047-bdf7-2da219e22717 nodeName:}" failed. No retries permitted until 2026-09-10 11:16:18.527906523 +0000 UTC m=+96.603232172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls") pod "node-exporter-vtwh4" (UID: "6b671393-bc0d-4047-bdf7-2da219e22717") : secret "node-exporter-tls" not found
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.027948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plhn\" (UniqueName: \"kubernetes.io/projected/6b671393-bc0d-4047-bdf7-2da219e22717-kube-api-access-5plhn\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.028007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-sys\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.028073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b671393-bc0d-4047-bdf7-2da219e22717-sys\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.028186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-textfile\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.028519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-metrics-client-ca\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.028651 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.028625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.030272 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.030254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.039191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.039165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.043580 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.043556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plhn\" (UniqueName: \"kubernetes.io/projected/6b671393-bc0d-4047-bdf7-2da219e22717-kube-api-access-5plhn\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.139068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.139039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.233182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.233149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.331788 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.331718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.532726 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.532692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.535070 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.535052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b671393-bc0d-4047-bdf7-2da219e22717-node-exporter-tls\") pod \"node-exporter-vtwh4\" (UID: \"6b671393-bc0d-4047-bdf7-2da219e22717\") " pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.671710 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.671676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.681860 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.681830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtwh4"
Sep 10 11:16:18.693252 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:18.693224    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b671393_bc0d_4047_bdf7_2da219e22717.slice/crio-fbb879e116241545f6379092e247dbcc93df5bc538aef5520fc5fe6b9bfe9576 WatchSource:0}: Error finding container fbb879e116241545f6379092e247dbcc93df5bc538aef5520fc5fe6b9bfe9576: Status 404 returned error can't find the container with id fbb879e116241545f6379092e247dbcc93df5bc538aef5520fc5fe6b9bfe9576
Sep 10 11:16:18.787559 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.787525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:18.858285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.858252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtwh4" event={"ID":"6b671393-bc0d-4047-bdf7-2da219e22717","Type":"ContainerStarted","Data":"fbb879e116241545f6379092e247dbcc93df5bc538aef5520fc5fe6b9bfe9576"}
Sep 10 11:16:18.899978 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:18.899948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:19.863974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:19.863940    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b671393-bc0d-4047-bdf7-2da219e22717" containerID="d0f26a86efd3b33945a22cfbcc453518097a3e870522165a2b32486f80809843" exitCode=0
Sep 10 11:16:19.864457 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:19.863999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtwh4" event={"ID":"6b671393-bc0d-4047-bdf7-2da219e22717","Type":"ContainerDied","Data":"d0f26a86efd3b33945a22cfbcc453518097a3e870522165a2b32486f80809843"}
Sep 10 11:16:20.450337 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.450308    2578 scope.go:117] "RemoveContainer" containerID="1dd69c04789d1482fa03cb33f054a8ae7eb9ad971f235dd63e18b820cd86e488"
Sep 10 11:16:20.692485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.692453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:20.731987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.731911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zczmj"
Sep 10 11:16:20.803681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.803643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:20.852888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.852839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:16:20.855419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.855396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e3b4414-485e-4bf7-8743-c0b0b6a47226-metrics-tls\") pod \"dns-default-dbfxf\" (UID: \"5e3b4414-485e-4bf7-8743-c0b0b6a47226\") " pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:16:20.870010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.869969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtwh4" event={"ID":"6b671393-bc0d-4047-bdf7-2da219e22717","Type":"ContainerStarted","Data":"e032607c1fd4b44c66ced45df2e916cbc6a3c773328aa00ce141c52eee588f47"}
Sep 10 11:16:20.870422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.870039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtwh4" event={"ID":"6b671393-bc0d-4047-bdf7-2da219e22717","Type":"ContainerStarted","Data":"3a2faff513c3a43ab9b17a335c06c7b612178c97cc4711155f69259b2896610f"}
Sep 10 11:16:20.871832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.871810    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:16:20.871926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.871884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-glrhj" event={"ID":"f59e35f6-4a05-4a79-86fd-f5aeaf409d61","Type":"ContainerStarted","Data":"0de73253cf36bb96600d5acc13592bfb1c44880de29ad3149c25fdd82fbd8677"}
Sep 10 11:16:20.872130 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.872114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:16:20.908256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.908200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vtwh4" podStartSLOduration=2.898535222 podStartE2EDuration="3.908182915s" totalImagesPullingTime="940.635401ms" totalInitContainerRuntime="69.012292ms" isStatefulPod=true podCreationTimestamp="2026-09-10 11:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:20.906980584 +0000 UTC m=+98.982306251" watchObservedRunningTime="2026-09-10 11:16:20.908182915 +0000 UTC m=+98.983508590"
Sep 10 11:16:20.930790 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:20.930739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-glrhj" podStartSLOduration=50.855759148 podStartE2EDuration="53.930723685s" totalImagesPullingTime="3.074964537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:20.928886139 +0000 UTC m=+99.004211803" watchObservedRunningTime="2026-09-10 11:16:20.930723685 +0000 UTC m=+99.006049402"
Sep 10 11:16:21.043174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.043065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-glrhj"
Sep 10 11:16:21.076374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.076337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-krm6r\""
Sep 10 11:16:21.083493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.083463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:16:21.386539 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.386461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbfxf"]
Sep 10 11:16:21.389157 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:21.389132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3b4414_485e_4bf7_8743_c0b0b6a47226.slice/crio-dff89d1bdec34a6ba993d1fcd6b1fcc843c994836136d7b8f817402f26f66c08 WatchSource:0}: Error finding container dff89d1bdec34a6ba993d1fcd6b1fcc843c994836136d7b8f817402f26f66c08: Status 404 returned error can't find the container with id dff89d1bdec34a6ba993d1fcd6b1fcc843c994836136d7b8f817402f26f66c08
Sep 10 11:16:21.401994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.401965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:21.561707 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.561662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:21.661841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.661751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:21.876721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.876687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbfxf" event={"ID":"5e3b4414-485e-4bf7-8743-c0b0b6a47226","Type":"ContainerStarted","Data":"dff89d1bdec34a6ba993d1fcd6b1fcc843c994836136d7b8f817402f26f66c08"}
Sep 10 11:16:21.923448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:21.923361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.013777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.013742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.143355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.143320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.250463 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.250347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.331536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.331499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.434953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.434919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.562055 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.561981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.662908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.662870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.759547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.759513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:22.980277 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:22.980238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:23.496039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:23.496008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:23.887001 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:23.886972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbfxf" event={"ID":"5e3b4414-485e-4bf7-8743-c0b0b6a47226","Type":"ContainerStarted","Data":"89e106ef6a372bc46f936b48f549dc67da0bc1fced3f8a1417bb35b9df93176b"}
Sep 10 11:16:23.887169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:23.887053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbfxf" event={"ID":"5e3b4414-485e-4bf7-8743-c0b0b6a47226","Type":"ContainerStarted","Data":"cb5d487109bb23c3e2382e915fe78c45bc9146d0b03dacb8bdca28a5f6eee053"}
Sep 10 11:16:23.887169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:23.887108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:16:23.916622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:23.916566    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbfxf" podStartSLOduration=65.856722934 podStartE2EDuration="1m7.916550099s" totalImagesPullingTime="2.059827165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:23.915434884 +0000 UTC m=+101.990760550" watchObservedRunningTime="2026-09-10 11:16:23.916550099 +0000 UTC m=+101.991875762"
Sep 10 11:16:24.001854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:24.001762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:24.125387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:24.125355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:24.281312 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:24.281234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:25.632169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:25.632137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:25.763708 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:25.763674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:27.258598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:27.258560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:27.359715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:27.359681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:28.420972 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:28.420938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:28.527725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:28.527690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:31.263390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:31.263354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:31.362883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:31.362849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:33.892814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:33.892788    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbfxf"
Sep 10 11:16:34.809797 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:34.809763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:34.935913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:34.935882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:36.983079 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:36.983046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:37.110761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:37.110721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:38.579356 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:38.579322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:38.721561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:38.721535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:38.849932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:38.849847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:39.236701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:39.236669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:39.333644 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:39.333608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:41.007323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.007293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:41.090588 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.090553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:41.541218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.541133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:41.623711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.623670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:41.950695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.950660    2578 generic.go:356] "Generic (PLEG): container finished" podID="2dc89937-563f-4148-a495-37312d5beea1" containerID="af4a7d1fce2910a3b80eb5efa72ab120f754b1ee1ecb09004d84d1b753820925" exitCode=0
Sep 10 11:16:41.950836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.950713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t" event={"ID":"2dc89937-563f-4148-a495-37312d5beea1","Type":"ContainerDied","Data":"af4a7d1fce2910a3b80eb5efa72ab120f754b1ee1ecb09004d84d1b753820925"}
Sep 10 11:16:41.951033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:41.951021    2578 scope.go:117] "RemoveContainer" containerID="af4a7d1fce2910a3b80eb5efa72ab120f754b1ee1ecb09004d84d1b753820925"
Sep 10 11:16:42.956429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:42.956394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8j76t" event={"ID":"2dc89937-563f-4148-a495-37312d5beea1","Type":"ContainerStarted","Data":"917915bd88716cd1a00257fea7c15f9fde01071819f476b42c94a9c28a38cd53"}
Sep 10 11:16:45.136245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:45.136213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:45.281289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:45.281257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:46.161673 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:46.161639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:46.232646 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:46.232616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:48.226243 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:48.226212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:49.136412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:49.136381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:49.236357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:49.236322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:50.598880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:50.598839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:50.726238 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:50.726205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:51.043247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:51.043208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:51.166910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:51.166863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:51.572559 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:51.572530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:51.668081 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:51.668047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:52.203872 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.203827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:16:52.206439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.206414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b95e13d-b128-4c10-b009-e54830d2be14-metrics-certs\") pod \"network-metrics-daemon-xmwgb\" (UID: \"8b95e13d-b128-4c10-b009-e54830d2be14\") " pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:16:52.365245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.365209    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9kdff\""
Sep 10 11:16:52.372208 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.372186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmwgb"
Sep 10 11:16:52.649492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.649467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmwgb"]
Sep 10 11:16:52.651896 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:16:52.651874    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b95e13d_b128_4c10_b009_e54830d2be14.slice/crio-e6b3faeb2d7d7b5aae87062ff226eb47cc0f6f5e42f769657474a48984c5796c WatchSource:0}: Error finding container e6b3faeb2d7d7b5aae87062ff226eb47cc0f6f5e42f769657474a48984c5796c: Status 404 returned error can't find the container with id e6b3faeb2d7d7b5aae87062ff226eb47cc0f6f5e42f769657474a48984c5796c
Sep 10 11:16:52.995913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:52.995827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmwgb" event={"ID":"8b95e13d-b128-4c10-b009-e54830d2be14","Type":"ContainerStarted","Data":"e6b3faeb2d7d7b5aae87062ff226eb47cc0f6f5e42f769657474a48984c5796c"}
Sep 10 11:16:54.005723 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:54.005682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmwgb" event={"ID":"8b95e13d-b128-4c10-b009-e54830d2be14","Type":"ContainerStarted","Data":"51f8f06c0f0ed8e2e15af7b3c7dc9a9f98831ca945bbadbc5c7cd0c5740d2d28"}
Sep 10 11:16:54.006165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:54.005748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmwgb" event={"ID":"8b95e13d-b128-4c10-b009-e54830d2be14","Type":"ContainerStarted","Data":"7522ea5f02d3ed5a3f134112283503c0f32a1a66a1d6b7eb2dee7f9b36297807"}
Sep 10 11:16:54.026135 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:54.026035    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xmwgb" podStartSLOduration=130.894207023 podStartE2EDuration="2m12.026011896s" totalImagesPullingTime="1.131804873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:16:54.024247269 +0000 UTC m=+132.099572934" watchObservedRunningTime="2026-09-10 11:16:54.026011896 +0000 UTC m=+132.101337564"
Sep 10 11:16:54.154219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:54.154181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:16:54.243230 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:16:54.243065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:00.211757 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:00.211726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:00.312413 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:00.312352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:13.291805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:13.291753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:13.419193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:13.419145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:14.792490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:14.792430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:33.947505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:33.947416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:34.041919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:34.041882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:45.858414 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:45.858373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:17:45.970520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:17:45.970486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:00.190391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:00.190360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:00.292373 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:00.292331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:11.739829 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:11.739790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:12.909858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:12.909823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:13.010322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:13.010288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:22.707734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:22.707700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:22.806371 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:22.806340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:22.901382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:22.901345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:32.872739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:32.872692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:33.023574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:33.023535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:55.978175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:55.978137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:57.430102 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:57.430053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:59.611258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:59.611217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:18:59.708205 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:18:59.708173    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:01.738548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:01.738514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:01.864394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:01.864364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:02.179498 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:02.179445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:02.359486 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:02.359455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:03.029517 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.029484    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"]
Sep 10 11:19:03.032740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.032724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.040783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.040761    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5lwg\""
Sep 10 11:19:03.040783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.040780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 11:19:03.042125 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.042108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 11:19:03.049062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.049040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"]
Sep 10 11:19:03.060302 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.060273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.060424 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.060314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7s9t\" (UniqueName: \"kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.060424 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.060335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.161572 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.161537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.161749 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.161593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.161749 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.161626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7s9t\" (UniqueName: \"kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.161957 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.161937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.162000 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.161957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.172381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.172353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7s9t\" (UniqueName: \"kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.341372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.341264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:03.620148 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:03.620122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"]
Sep 10 11:19:03.622771 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:19:03.622734    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4497f50_367e_4f86_bf4c_4cb43462086b.slice/crio-f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3 WatchSource:0}: Error finding container f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3: Status 404 returned error can't find the container with id f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3
Sep 10 11:19:04.204311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:04.204275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:04.461071 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:04.460987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp" event={"ID":"d4497f50-367e-4f86-bf4c-4cb43462086b","Type":"ContainerStarted","Data":"f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3"}
Sep 10 11:19:04.495415 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:04.495381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:06.804983 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:06.804953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:06.904250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:06.904218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:10.483660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:10.483621    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerID="899d066b01aa23f48205776f7258240f43bd136e2f493448767938ea89d964f3" exitCode=0
Sep 10 11:19:10.484017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:10.483677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp" event={"ID":"d4497f50-367e-4f86-bf4c-4cb43462086b","Type":"ContainerDied","Data":"899d066b01aa23f48205776f7258240f43bd136e2f493448767938ea89d964f3"}
Sep 10 11:19:11.776561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:11.776531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:14.125008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:14.124973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:14.222483 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:14.222444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:14.885298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:14.885265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:19.515876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:19.515782    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerID="c0782eacb8253dbdf09708887d4f6ad2e29441f5cda966496c5cf5b365b389ed" exitCode=0
Sep 10 11:19:19.515876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:19.515837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp" event={"ID":"d4497f50-367e-4f86-bf4c-4cb43462086b","Type":"ContainerDied","Data":"c0782eacb8253dbdf09708887d4f6ad2e29441f5cda966496c5cf5b365b389ed"}
Sep 10 11:19:27.550615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:27.550581    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerID="37ff2e22dfc5fdcd2345ccf22263f462189c41a02d82218d4454cdfb852c72ee" exitCode=0
Sep 10 11:19:27.551111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:27.550666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp" event={"ID":"d4497f50-367e-4f86-bf4c-4cb43462086b","Type":"ContainerDied","Data":"37ff2e22dfc5fdcd2345ccf22263f462189c41a02d82218d4454cdfb852c72ee"}
Sep 10 11:19:28.681369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.681348    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:28.771002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.770968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7s9t\" (UniqueName: \"kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t\") pod \"d4497f50-367e-4f86-bf4c-4cb43462086b\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") "
Sep 10 11:19:28.771002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.771008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util\") pod \"d4497f50-367e-4f86-bf4c-4cb43462086b\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") "
Sep 10 11:19:28.771276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.771050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle\") pod \"d4497f50-367e-4f86-bf4c-4cb43462086b\" (UID: \"d4497f50-367e-4f86-bf4c-4cb43462086b\") "
Sep 10 11:19:28.771449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.771425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle" (OuterVolumeSpecName: "bundle") pod "d4497f50-367e-4f86-bf4c-4cb43462086b" (UID: "d4497f50-367e-4f86-bf4c-4cb43462086b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:19:28.773358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.773332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t" (OuterVolumeSpecName: "kube-api-access-w7s9t") pod "d4497f50-367e-4f86-bf4c-4cb43462086b" (UID: "d4497f50-367e-4f86-bf4c-4cb43462086b"). InnerVolumeSpecName "kube-api-access-w7s9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:19:28.775132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.775109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util" (OuterVolumeSpecName: "util") pod "d4497f50-367e-4f86-bf4c-4cb43462086b" (UID: "d4497f50-367e-4f86-bf4c-4cb43462086b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:19:28.872467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.872440    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-bundle\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:19:28.872467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.872467    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7s9t\" (UniqueName: \"kubernetes.io/projected/d4497f50-367e-4f86-bf4c-4cb43462086b-kube-api-access-w7s9t\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:19:28.872641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:28.872476    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4497f50-367e-4f86-bf4c-4cb43462086b-util\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:19:29.559926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:29.559891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp" event={"ID":"d4497f50-367e-4f86-bf4c-4cb43462086b","Type":"ContainerDied","Data":"f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3"}
Sep 10 11:19:29.559926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:29.559911    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffz8rp"
Sep 10 11:19:29.559926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:29.559932    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c9fcfb578acadf5183ff00c775467ccd2ac2a83713b44f82527b9aa50213f3"
Sep 10 11:19:31.710194 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:31.710158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:31.806288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:31.806256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:33.721267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721232    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"]
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721469    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="pull"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721480    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="pull"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721491    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="extract"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721496    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="extract"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721505    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="util"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721510    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="util"
Sep 10 11:19:33.721678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.721559    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4497f50-367e-4f86-bf4c-4cb43462086b" containerName="extract"
Sep 10 11:19:33.724345 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.724329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.727383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.727363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:19:33.727489 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.727396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 11:19:33.728938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.728923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pzxgx\""
Sep 10 11:19:33.733911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.733888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"]
Sep 10 11:19:33.807626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.807585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28d8\" (UniqueName: \"kubernetes.io/projected/79ba8d4f-36db-4b78-862a-6db07c559f8e-kube-api-access-n28d8\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.807798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.807676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ba8d4f-36db-4b78-862a-6db07c559f8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.908357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.908324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ba8d4f-36db-4b78-862a-6db07c559f8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.908545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.908367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28d8\" (UniqueName: \"kubernetes.io/projected/79ba8d4f-36db-4b78-862a-6db07c559f8e-kube-api-access-n28d8\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.908749 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.908725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ba8d4f-36db-4b78-862a-6db07c559f8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:33.919039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:33.919015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28d8\" (UniqueName: \"kubernetes.io/projected/79ba8d4f-36db-4b78-862a-6db07c559f8e-kube-api-access-n28d8\") pod \"openshift-lws-operator-fd8ccff4c-hvrkv\" (UID: \"79ba8d4f-36db-4b78-862a-6db07c559f8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:34.033519 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:34.033421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"
Sep 10 11:19:34.304631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:34.304583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv"]
Sep 10 11:19:34.307168 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:19:34.307135    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79ba8d4f_36db_4b78_862a_6db07c559f8e.slice/crio-5b2f0d08193344ffafe685a52f1a0191849ee18be3405fcf0f8942870b3c34fe WatchSource:0}: Error finding container 5b2f0d08193344ffafe685a52f1a0191849ee18be3405fcf0f8942870b3c34fe: Status 404 returned error can't find the container with id 5b2f0d08193344ffafe685a52f1a0191849ee18be3405fcf0f8942870b3c34fe
Sep 10 11:19:34.583322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:34.583239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv" event={"ID":"79ba8d4f-36db-4b78-862a-6db07c559f8e","Type":"ContainerStarted","Data":"5b2f0d08193344ffafe685a52f1a0191849ee18be3405fcf0f8942870b3c34fe"}
Sep 10 11:19:36.592500 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:36.592464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv" event={"ID":"79ba8d4f-36db-4b78-862a-6db07c559f8e","Type":"ContainerStarted","Data":"5c010e0af98ecbbc30a589acca0c45f48c40c4957ab0b602677f6699c3325d8b"}
Sep 10 11:19:36.615384 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:36.615336    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hvrkv" podStartSLOduration=1.869693649 podStartE2EDuration="3.615320912s" totalImagesPullingTime="1.745627263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:19:36.613872258 +0000 UTC m=+294.689197924" watchObservedRunningTime="2026-09-10 11:19:36.615320912 +0000 UTC m=+294.690646576"
Sep 10 11:19:37.345419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:37.345385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:37.447416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:37.447385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:39.539865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:39.539829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:42.352253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:42.352218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:19:42.353165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:42.353135    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:19:42.365174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:42.365150    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 11:19:43.807297 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.807266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:43.903131 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.903077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:43.920571 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.920539    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"]
Sep 10 11:19:43.923459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.923442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:43.928322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.928297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 11:19:43.928470 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.928297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-789bb\""
Sep 10 11:19:43.930549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.930532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 11:19:43.930618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.930584    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 11:19:43.955323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:43.955290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"]
Sep 10 11:19:44.091212 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.091116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3784c54-65cb-49f8-af70-37fbbcffce9b-manager-config\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.091379 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.091240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4h2\" (UniqueName: \"kubernetes.io/projected/e3784c54-65cb-49f8-af70-37fbbcffce9b-kube-api-access-kb4h2\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.091379 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.091272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-metrics-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.091379 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.091291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.192634 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.192581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-metrics-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.192634 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.192640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.192855 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.192693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3784c54-65cb-49f8-af70-37fbbcffce9b-manager-config\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.192855 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.192759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4h2\" (UniqueName: \"kubernetes.io/projected/e3784c54-65cb-49f8-af70-37fbbcffce9b-kube-api-access-kb4h2\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.193420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.193400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3784c54-65cb-49f8-af70-37fbbcffce9b-manager-config\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.195353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.195321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.195463 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.195407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3784c54-65cb-49f8-af70-37fbbcffce9b-metrics-cert\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.202554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.202527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4h2\" (UniqueName: \"kubernetes.io/projected/e3784c54-65cb-49f8-af70-37fbbcffce9b-kube-api-access-kb4h2\") pod \"lws-controller-manager-7b89dcdb5f-4bnj2\" (UID: \"e3784c54-65cb-49f8-af70-37fbbcffce9b\") " pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.232908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.232873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:44.505866 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.505836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"]
Sep 10 11:19:44.508901 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:19:44.508876    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3784c54_65cb_49f8_af70_37fbbcffce9b.slice/crio-b6822782be3710a088e5a0fe6e8a32224844c994975e8de3b182c607d5c720cf WatchSource:0}: Error finding container b6822782be3710a088e5a0fe6e8a32224844c994975e8de3b182c607d5c720cf: Status 404 returned error can't find the container with id b6822782be3710a088e5a0fe6e8a32224844c994975e8de3b182c607d5c720cf
Sep 10 11:19:44.510839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.510822    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:19:44.620442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:44.620405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2" event={"ID":"e3784c54-65cb-49f8-af70-37fbbcffce9b","Type":"ContainerStarted","Data":"b6822782be3710a088e5a0fe6e8a32224844c994975e8de3b182c607d5c720cf"}
Sep 10 11:19:46.629816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:46.629765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2" event={"ID":"e3784c54-65cb-49f8-af70-37fbbcffce9b","Type":"ContainerStarted","Data":"c3b019b0dd06ffadda2aaf2b9dbd6a8c8eb5cce18b3390eaaef8e7d7d6310e0c"}
Sep 10 11:19:46.630241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:46.629895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:46.649142 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:46.649070    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2" podStartSLOduration=2.1737938 podStartE2EDuration="3.649053266s" totalImagesPullingTime="1.475259466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:19:46.648386816 +0000 UTC m=+304.723712484" watchObservedRunningTime="2026-09-10 11:19:46.649053266 +0000 UTC m=+304.724378930"
Sep 10 11:19:55.258404 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:55.258369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:55.348583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:55.348553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:55.861948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:55.861915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.263875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.263752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.356097 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.356050    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"]
Sep 10 11:19:56.361909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.361891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.367982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.367956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 11:19:56.368121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.367965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 11:19:56.368688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.368670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zxnsf\""
Sep 10 11:19:56.368775 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.368673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 11:19:56.368880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.368861    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 11:19:56.378537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.378511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vf9\" (UniqueName: \"kubernetes.io/projected/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-kube-api-access-f9vf9\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.378643 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.378540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-tmp\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.378643 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.378602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.378751 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.378645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-manifests\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.378751 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.378677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.381523 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.381500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"]
Sep 10 11:19:56.384282 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.384261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.479713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.479682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vf9\" (UniqueName: \"kubernetes.io/projected/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-kube-api-access-f9vf9\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.479713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.479715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-tmp\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.479951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.479774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.479951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.479802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-manifests\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.479951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.479817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.480257 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.480239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-manifests\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.482221 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.482196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-tmp\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.482421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.482401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.482550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.482530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.491328 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.491308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vf9\" (UniqueName: \"kubernetes.io/projected/5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5-kube-api-access-f9vf9\") pod \"opendatahub-operator-controller-manager-d6d6998d4-nqd6s\" (UID: \"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5\") " pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.535352 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.535233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.655164 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.655077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.672393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.672372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:19:56.756695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.756657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.862772 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.862741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.965733 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.965707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"]
Sep 10 11:19:56.967520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:56.967492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:19:56.968673 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:19:56.968653    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd4ba6f_d8a1_4c8f_9f5c_5f414d3ce1e5.slice/crio-79fe53a7db613b93903ab8bdab3197506121efa70d91a457ad71448ee3b78b4e WatchSource:0}: Error finding container 79fe53a7db613b93903ab8bdab3197506121efa70d91a457ad71448ee3b78b4e: Status 404 returned error can't find the container with id 79fe53a7db613b93903ab8bdab3197506121efa70d91a457ad71448ee3b78b4e
Sep 10 11:19:57.637261 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:57.637230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b89dcdb5f-4bnj2"
Sep 10 11:19:57.676535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:19:57.676499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s" event={"ID":"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5","Type":"ContainerStarted","Data":"79fe53a7db613b93903ab8bdab3197506121efa70d91a457ad71448ee3b78b4e"}
Sep 10 11:20:01.696812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:01.696776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s" event={"ID":"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5","Type":"ContainerStarted","Data":"c2a762ada3d0a647a6c9aabe0b898078fcfa68020fc56d9f56bb57bf530723c6"}
Sep 10 11:20:02.701892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:02.701810    2578 generic.go:356] "Generic (PLEG): container finished" podID="5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5" containerID="c2a762ada3d0a647a6c9aabe0b898078fcfa68020fc56d9f56bb57bf530723c6" exitCode=0
Sep 10 11:20:02.701892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:02.701886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s" event={"ID":"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5","Type":"ContainerDied","Data":"c2a762ada3d0a647a6c9aabe0b898078fcfa68020fc56d9f56bb57bf530723c6"}
Sep 10 11:20:03.709808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:03.709720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s" event={"ID":"5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5","Type":"ContainerStarted","Data":"0ba438ffbeb6fcf50ff6e32cb1b8d39b24613069b1f620e3fdccc8831af42dee"}
Sep 10 11:20:03.710225 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:03.709830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:20:03.751460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:03.751393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s" podStartSLOduration=1.815861123 podStartE2EDuration="7.751372541s" totalImagesPullingTime="4.89806248s" totalInitContainerRuntime="1.037448938s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:19:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:20:03.74900113 +0000 UTC m=+321.824326793" watchObservedRunningTime="2026-09-10 11:20:03.751372541 +0000 UTC m=+321.826698206"
Sep 10 11:20:14.716805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:14.716774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d6d6998d4-nqd6s"
Sep 10 11:20:16.480068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:16.480030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:18.563179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:18.563148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:27.432970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:27.432935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:27.601434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:27.601392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:27.706901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:27.706820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:27.803064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:27.803029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:28.045015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:28.044932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:28.264230 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:28.264200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:28.621719 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:28.621692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:29.251348 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:29.251310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:29.392918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:29.392852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:30.735704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:30.735671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:31.241938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:31.241904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:31.345169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:31.345138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:36.120124 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.120037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:36.280795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.280761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:36.619726 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.619699    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"]
Sep 10 11:20:36.627909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.627888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.633699 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.633676    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 11:20:36.633699 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.633696    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 11:20:36.643104 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.643039    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 11:20:36.643235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.643080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-m95nt\""
Sep 10 11:20:36.683779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsfc\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-kube-api-access-xbsfc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.683779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.683982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.683982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.683982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.683982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.683949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.684179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.684010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.684179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.684040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.684179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.684073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.697650 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.697621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"]
Sep 10 11:20:36.775321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.775293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:36.784475 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsfc\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-kube-api-access-xbsfc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.784950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.784941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.785059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.785024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.785232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.785210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.785601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.785583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.787613 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.787596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.787710 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.787692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.853403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.853367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsfc\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-kube-api-access-xbsfc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.862126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.862099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6\" (UID: \"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:36.943987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:36.943886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:37.127631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:37.127601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:37.301457 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:37.301434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"]
Sep 10 11:20:37.304538 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:20:37.304512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7d4419_a9e1_47f6_ac98_5a87fb6b3ee4.slice/crio-2c2912475c9a062efd004e21be8e8c02921e2c32908c0a20cfc77131bf1247a9 WatchSource:0}: Error finding container 2c2912475c9a062efd004e21be8e8c02921e2c32908c0a20cfc77131bf1247a9: Status 404 returned error can't find the container with id 2c2912475c9a062efd004e21be8e8c02921e2c32908c0a20cfc77131bf1247a9
Sep 10 11:20:37.393815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:37.393781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:37.843334 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:37.843296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6" event={"ID":"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4","Type":"ContainerStarted","Data":"2c2912475c9a062efd004e21be8e8c02921e2c32908c0a20cfc77131bf1247a9"}
Sep 10 11:20:39.928834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:39.928789    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:20:39.929156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:39.928865    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:20:39.929156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:39.928893    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:20:40.857998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:40.857968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6" event={"ID":"fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4","Type":"ContainerStarted","Data":"41977e10d294831d8c016b8bd208da74e7bd745dad27a7768e57997b40311ac6"}
Sep 10 11:20:40.944336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:40.944300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:40.948963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:40.948939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:40.971575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:40.971525    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6" podStartSLOduration=2.349483691 podStartE2EDuration="4.971507705s" totalImagesPullingTime="2.622024014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:20:40.965051213 +0000 UTC m=+359.040376877" watchObservedRunningTime="2026-09-10 11:20:40.971507705 +0000 UTC m=+359.046833369"
Sep 10 11:20:41.862568 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:41.862534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:41.863539 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:41.863520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6"
Sep 10 11:20:43.885223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:43.885190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:44.100154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.100120    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"]
Sep 10 11:20:44.104748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.104733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.116504 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.116484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 11:20:44.116633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.116617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-g9t7b\""
Sep 10 11:20:44.176492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.176420    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"]
Sep 10 11:20:44.217423 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.217389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:44.254149 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.254114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c100be0-eb1e-402a-914b-be6286228675-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.254309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.254161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kdl\" (UniqueName: \"kubernetes.io/projected/1c100be0-eb1e-402a-914b-be6286228675-kube-api-access-q8kdl\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.355003 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.354963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c100be0-eb1e-402a-914b-be6286228675-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.355003 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.355010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kdl\" (UniqueName: \"kubernetes.io/projected/1c100be0-eb1e-402a-914b-be6286228675-kube-api-access-q8kdl\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.357588 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.357554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c100be0-eb1e-402a-914b-be6286228675-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.374060 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.374036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kdl\" (UniqueName: \"kubernetes.io/projected/1c100be0-eb1e-402a-914b-be6286228675-kube-api-access-q8kdl\") pod \"llmisvc-controller-manager-777c6bd7d5-m9c44\" (UID: \"1c100be0-eb1e-402a-914b-be6286228675\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.413823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.413787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:44.520393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.520364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:44.698374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.698336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"]
Sep 10 11:20:44.703121 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:20:44.703067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c100be0_eb1e_402a_914b_be6286228675.slice/crio-1687ab7378a8c55c27664cd784cdb85685fcde05ccbacb5f870da35bb4d5a45b WatchSource:0}: Error finding container 1687ab7378a8c55c27664cd784cdb85685fcde05ccbacb5f870da35bb4d5a45b: Status 404 returned error can't find the container with id 1687ab7378a8c55c27664cd784cdb85685fcde05ccbacb5f870da35bb4d5a45b
Sep 10 11:20:44.721114 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.721049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:44.876064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:44.876031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44" event={"ID":"1c100be0-eb1e-402a-914b-be6286228675","Type":"ContainerStarted","Data":"1687ab7378a8c55c27664cd784cdb85685fcde05ccbacb5f870da35bb4d5a45b"}
Sep 10 11:20:47.258818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:47.258778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:47.540482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:47.540391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:20:47.893247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:47.893220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44" event={"ID":"1c100be0-eb1e-402a-914b-be6286228675","Type":"ContainerStarted","Data":"62d43ca7e6910455328c8025c758f5f09136ede339afcd080c65e77c5a551d7e"}
Sep 10 11:20:47.893421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:47.893357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:20:47.939721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:47.939667    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44" podStartSLOduration=1.203911325 podStartE2EDuration="3.939651479s" totalImagesPullingTime="2.735740154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:20:47.939140939 +0000 UTC m=+366.014466606" watchObservedRunningTime="2026-09-10 11:20:47.939651479 +0000 UTC m=+366.014977154"
Sep 10 11:20:57.096411 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:20:57.096372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:18.899537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:18.899508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m9c44"
Sep 10 11:21:29.039159 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:29.039122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:29.140859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:29.140829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:34.007526 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.007490    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"]
Sep 10 11:21:34.011287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.011265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.014570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.014552    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 11:21:34.014686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.014607    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 11:21:34.016014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.015998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5lwg\""
Sep 10 11:21:34.021377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.021356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"]
Sep 10 11:21:34.052108 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.052049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.052108 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.052109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljp54\" (UniqueName: \"kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.052313 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.052194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.104841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.104803    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"]
Sep 10 11:21:34.108274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.108258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.118715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.118688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"]
Sep 10 11:21:34.153008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.152970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghn7\" (UniqueName: \"kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.153196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.153196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.153196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljp54\" (UniqueName: \"kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.153196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.153339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.153529 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.153564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.153535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.164554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.164525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljp54\" (UniqueName: \"kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.254434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.254395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghn7\" (UniqueName: \"kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.254434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.254442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.254644 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.254478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.254835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.254819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.254882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.254863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.264698 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.264631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghn7\" (UniqueName: \"kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.320442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.320409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:34.416410 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.416379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:34.594460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.594433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"]
Sep 10 11:21:34.596483 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:21:34.596454    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f55b2a1_34fd_459c_84ae_5a8f2661cd2e.slice/crio-ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c WatchSource:0}: Error finding container ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c: Status 404 returned error can't find the container with id ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c
Sep 10 11:21:34.691599 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:34.691474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"]
Sep 10 11:21:34.693817 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:21:34.693790    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f40bc9_ec57_49e3_b505_566c0bf5eefd.slice/crio-6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f WatchSource:0}: Error finding container 6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f: Status 404 returned error can't find the container with id 6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f
Sep 10 11:21:35.085387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.085347    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerID="6e894fd68001244e0147a10ac71d73de855bce842cb72b12ea7e98046f7577c6" exitCode=0
Sep 10 11:21:35.085802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.085428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c" event={"ID":"f0f40bc9-ec57-49e3-b505-566c0bf5eefd","Type":"ContainerDied","Data":"6e894fd68001244e0147a10ac71d73de855bce842cb72b12ea7e98046f7577c6"}
Sep 10 11:21:35.085802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.085478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c" event={"ID":"f0f40bc9-ec57-49e3-b505-566c0bf5eefd","Type":"ContainerStarted","Data":"6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f"}
Sep 10 11:21:35.086975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.086840    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerID="ff69e498499e42cafc1c0e72883309b099722b7fb20accb2144d58ef6fc1e031" exitCode=0
Sep 10 11:21:35.086975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.086864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs" event={"ID":"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e","Type":"ContainerDied","Data":"ff69e498499e42cafc1c0e72883309b099722b7fb20accb2144d58ef6fc1e031"}
Sep 10 11:21:35.086975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.086888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs" event={"ID":"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e","Type":"ContainerStarted","Data":"ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c"}
Sep 10 11:21:35.325390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.325363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:35.437987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:35.437952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:36.100575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:36.100549    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerID="7bf01ba13c2b61b63f26e79f2657e8a37a9670b5144a36b89a379a41e31eb71c" exitCode=0
Sep 10 11:21:36.100945 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:36.100625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs" event={"ID":"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e","Type":"ContainerDied","Data":"7bf01ba13c2b61b63f26e79f2657e8a37a9670b5144a36b89a379a41e31eb71c"}
Sep 10 11:21:37.109300 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:37.109264    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerID="c19951b4b9798713e988d076992758a3f1ab79b7f394236e50337bdb30b3580b" exitCode=0
Sep 10 11:21:37.109736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:37.109347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs" event={"ID":"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e","Type":"ContainerDied","Data":"c19951b4b9798713e988d076992758a3f1ab79b7f394236e50337bdb30b3580b"}
Sep 10 11:21:37.111141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:37.111120    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerID="1f39577acbfe2d705ef80bebc44f21ea0c82fd3eab0dde86537f14329834fe1a" exitCode=0
Sep 10 11:21:37.111247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:37.111162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c" event={"ID":"f0f40bc9-ec57-49e3-b505-566c0bf5eefd","Type":"ContainerDied","Data":"1f39577acbfe2d705ef80bebc44f21ea0c82fd3eab0dde86537f14329834fe1a"}
Sep 10 11:21:38.117758 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.117725    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerID="fd0b97c79db4c302f6d962a5545a3d1df6e41b8b99f6cc50f9c53386a67260b9" exitCode=0
Sep 10 11:21:38.118229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.117804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c" event={"ID":"f0f40bc9-ec57-49e3-b505-566c0bf5eefd","Type":"ContainerDied","Data":"fd0b97c79db4c302f6d962a5545a3d1df6e41b8b99f6cc50f9c53386a67260b9"}
Sep 10 11:21:38.242196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.242175    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:38.287192 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.287162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljp54\" (UniqueName: \"kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54\") pod \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") "
Sep 10 11:21:38.287360 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.287206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle\") pod \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") "
Sep 10 11:21:38.287360 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.287343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util\") pod \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\" (UID: \"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e\") "
Sep 10 11:21:38.287785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.287755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle" (OuterVolumeSpecName: "bundle") pod "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" (UID: "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:21:38.289518 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.289493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54" (OuterVolumeSpecName: "kube-api-access-ljp54") pod "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" (UID: "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e"). InnerVolumeSpecName "kube-api-access-ljp54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:21:38.293005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.292984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util" (OuterVolumeSpecName: "util") pod "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" (UID: "4f55b2a1-34fd-459c-84ae-5a8f2661cd2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:21:38.388712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.388615    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-util\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:38.388712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.388659    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljp54\" (UniqueName: \"kubernetes.io/projected/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-kube-api-access-ljp54\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:38.388712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:38.388670    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f55b2a1-34fd-459c-84ae-5a8f2661cd2e-bundle\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:39.124062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.124030    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs"
Sep 10 11:21:39.124062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.124054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n7fcs" event={"ID":"4f55b2a1-34fd-459c-84ae-5a8f2661cd2e","Type":"ContainerDied","Data":"ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c"}
Sep 10 11:21:39.124543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.124119    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe4cec8f337e6b8d1f8c5de203961204a39dc5104cd543dbe726108914f601c"
Sep 10 11:21:39.254373 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.254349    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:39.296704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.296678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mghn7\" (UniqueName: \"kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7\") pod \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") "
Sep 10 11:21:39.296845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.296755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle\") pod \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") "
Sep 10 11:21:39.296845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.296787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util\") pod \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\" (UID: \"f0f40bc9-ec57-49e3-b505-566c0bf5eefd\") "
Sep 10 11:21:39.297499 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.297467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle" (OuterVolumeSpecName: "bundle") pod "f0f40bc9-ec57-49e3-b505-566c0bf5eefd" (UID: "f0f40bc9-ec57-49e3-b505-566c0bf5eefd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:21:39.298904 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.298884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7" (OuterVolumeSpecName: "kube-api-access-mghn7") pod "f0f40bc9-ec57-49e3-b505-566c0bf5eefd" (UID: "f0f40bc9-ec57-49e3-b505-566c0bf5eefd"). InnerVolumeSpecName "kube-api-access-mghn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:21:39.302326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.302299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util" (OuterVolumeSpecName: "util") pod "f0f40bc9-ec57-49e3-b505-566c0bf5eefd" (UID: "f0f40bc9-ec57-49e3-b505-566c0bf5eefd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:21:39.398366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.398276    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-bundle\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:39.398366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.398308    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-util\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:39.398366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:39.398320    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghn7\" (UniqueName: \"kubernetes.io/projected/f0f40bc9-ec57-49e3-b505-566c0bf5eefd-kube-api-access-mghn7\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:21:40.130052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:40.130016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c" event={"ID":"f0f40bc9-ec57-49e3-b505-566c0bf5eefd","Type":"ContainerDied","Data":"6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f"}
Sep 10 11:21:40.130052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:40.130049    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a85q4c"
Sep 10 11:21:40.130512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:40.130107    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c4e75e52bb9212f09b6c0b9cb854f58e55c417fb81816ab826e5f188207075f"
Sep 10 11:21:43.596723 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:43.596689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:44.389128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:44.389072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:46.711602 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.711568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:46.808667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.808632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:46.928690 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.928659    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6lv9g"]
Sep 10 11:21:46.928971 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.928958    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="util"
Sep 10 11:21:46.929014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.928976    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="util"
Sep 10 11:21:46.929014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.928987    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="extract"
Sep 10 11:21:46.929014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.928993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="extract"
Sep 10 11:21:46.929014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929003    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="pull"
Sep 10 11:21:46.929014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929008    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="pull"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929018    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="extract"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929023    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="extract"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929034    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="pull"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929039    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="pull"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929049    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="util"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929054    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="util"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f55b2a1-34fd-459c-84ae-5a8f2661cd2e" containerName="extract"
Sep 10 11:21:46.929204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.929128    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f40bc9-ec57-49e3-b505-566c0bf5eefd" containerName="extract"
Sep 10 11:21:46.933137 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.933120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:46.939615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.939282    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 11:21:46.939896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.939873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xq69z\""
Sep 10 11:21:46.940800 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.940785    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 11:21:46.966964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.966890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6lv9g"]
Sep 10 11:21:46.997996 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:46.997968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:47.061037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.061006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmt7\" (UniqueName: \"kubernetes.io/projected/d0ebd4e8-ba22-4c92-bf2b-2987c2516708-kube-api-access-bgmt7\") pod \"authorino-operator-76b478bd78-6lv9g\" (UID: \"d0ebd4e8-ba22-4c92-bf2b-2987c2516708\") " pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:47.162399 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.162365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmt7\" (UniqueName: \"kubernetes.io/projected/d0ebd4e8-ba22-4c92-bf2b-2987c2516708-kube-api-access-bgmt7\") pod \"authorino-operator-76b478bd78-6lv9g\" (UID: \"d0ebd4e8-ba22-4c92-bf2b-2987c2516708\") " pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:47.174966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.174936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmt7\" (UniqueName: \"kubernetes.io/projected/d0ebd4e8-ba22-4c92-bf2b-2987c2516708-kube-api-access-bgmt7\") pod \"authorino-operator-76b478bd78-6lv9g\" (UID: \"d0ebd4e8-ba22-4c92-bf2b-2987c2516708\") " pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:47.190730 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.190697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:47.242894 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.242812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:47.492077 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.492040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:47.532621 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.532596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6lv9g"]
Sep 10 11:21:47.534867 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:21:47.534835    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ebd4e8_ba22_4c92_bf2b_2987c2516708.slice/crio-a3a92d58ced6d32266605f4cca3f0ea5aa86bd568434a2b0b5bb8ee91d491ec1 WatchSource:0}: Error finding container a3a92d58ced6d32266605f4cca3f0ea5aa86bd568434a2b0b5bb8ee91d491ec1: Status 404 returned error can't find the container with id a3a92d58ced6d32266605f4cca3f0ea5aa86bd568434a2b0b5bb8ee91d491ec1
Sep 10 11:21:47.612299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:47.612270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:48.038576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:48.038545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:48.164368 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:48.164337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g" event={"ID":"d0ebd4e8-ba22-4c92-bf2b-2987c2516708","Type":"ContainerStarted","Data":"a3a92d58ced6d32266605f4cca3f0ea5aa86bd568434a2b0b5bb8ee91d491ec1"}
Sep 10 11:21:48.404605 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:48.404573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:48.808969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:48.808889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:49.307669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:49.307635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:49.426882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:49.426841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:50.174843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:50.174748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g" event={"ID":"d0ebd4e8-ba22-4c92-bf2b-2987c2516708","Type":"ContainerStarted","Data":"4258a9490c80769f79b84cbf9b0d438f3d44a572251d7f831f6034fa45363d50"}
Sep 10 11:21:50.175016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:50.174880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:21:50.198038 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:50.197989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g" podStartSLOduration=1.851131477 podStartE2EDuration="4.197972989s" totalImagesPullingTime="2.346841512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:21:50.196362157 +0000 UTC m=+428.271687835" watchObservedRunningTime="2026-09-10 11:21:50.197972989 +0000 UTC m=+428.273298652"
Sep 10 11:21:52.417226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:52.417190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:53.879394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:53.879367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:54.600443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:54.600411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:55.310117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:55.310059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:57.622672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:57.622636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:57.836255 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:57.836221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:58.029930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:58.029849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:58.133236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:58.133202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:59.227937 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.227904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:59.240364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.240338    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"]
Sep 10 11:21:59.243506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.243491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.246640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.246618    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-75ljd\""
Sep 10 11:21:59.247071 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.247052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 11:21:59.247237 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.247221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 11:21:59.252901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.252880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"]
Sep 10 11:21:59.360552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.360513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gb44\" (UniqueName: \"kubernetes.io/projected/477c6af0-0e3d-4fde-90ab-f884dfa16d92-kube-api-access-9gb44\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.360736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.360594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/477c6af0-0e3d-4fde-90ab-f884dfa16d92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.360736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.360615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.461289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.461251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.461467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.461305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gb44\" (UniqueName: \"kubernetes.io/projected/477c6af0-0e3d-4fde-90ab-f884dfa16d92-kube-api-access-9gb44\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.461467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.461360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/477c6af0-0e3d-4fde-90ab-f884dfa16d92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.461467 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:21:59.461406    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 11:21:59.461592 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:21:59.461482    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert podName:477c6af0-0e3d-4fde-90ab-f884dfa16d92 nodeName:}" failed. No retries permitted until 2026-09-10 11:21:59.961467016 +0000 UTC m=+438.036792663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-4kzpg" (UID: "477c6af0-0e3d-4fde-90ab-f884dfa16d92") : secret "plugin-serving-cert" not found
Sep 10 11:21:59.461973 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.461956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/477c6af0-0e3d-4fde-90ab-f884dfa16d92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.472550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.472525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gb44\" (UniqueName: \"kubernetes.io/projected/477c6af0-0e3d-4fde-90ab-f884dfa16d92-kube-api-access-9gb44\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.696041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.695997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:21:59.965958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.965864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:21:59.968401 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:21:59.968382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/477c6af0-0e3d-4fde-90ab-f884dfa16d92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4kzpg\" (UID: \"477c6af0-0e3d-4fde-90ab-f884dfa16d92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:22:00.152907 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:00.152873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"
Sep 10 11:22:00.429291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:00.429258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg"]
Sep 10 11:22:00.432596 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:22:00.432562    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod477c6af0_0e3d_4fde_90ab_f884dfa16d92.slice/crio-ce1f42779cfa32e6a2852fb36a1e8d49a6a1ec4c8408940bc148c4ba76e7eaf2 WatchSource:0}: Error finding container ce1f42779cfa32e6a2852fb36a1e8d49a6a1ec4c8408940bc148c4ba76e7eaf2: Status 404 returned error can't find the container with id ce1f42779cfa32e6a2852fb36a1e8d49a6a1ec4c8408940bc148c4ba76e7eaf2
Sep 10 11:22:01.182322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:01.182288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-6lv9g"
Sep 10 11:22:01.225900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:01.225866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg" event={"ID":"477c6af0-0e3d-4fde-90ab-f884dfa16d92","Type":"ContainerStarted","Data":"ce1f42779cfa32e6a2852fb36a1e8d49a6a1ec4c8408940bc148c4ba76e7eaf2"}
Sep 10 11:22:03.410049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.409966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:03.486709 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.486675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:03.600299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.600266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:03.700806 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.700724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:03.801663 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.801622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:03.883482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:03.883445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:05.253189 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:05.253157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg" event={"ID":"477c6af0-0e3d-4fde-90ab-f884dfa16d92","Type":"ContainerStarted","Data":"ded732c455c795b5ed2c245c7fe2d0cd1be1731f441ceda3fdb447c5e8f1eb73"}
Sep 10 11:22:05.271974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:05.271930    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4kzpg" podStartSLOduration=1.670026618 podStartE2EDuration="6.271915771s" totalImagesPullingTime="4.601889153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:22:05.271752811 +0000 UTC m=+443.347078475" watchObservedRunningTime="2026-09-10 11:22:05.271915771 +0000 UTC m=+443.347241446"
Sep 10 11:22:09.969899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:09.969866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:10.130774 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:10.130739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:14.667216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:14.667185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:14.829017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:14.828983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:18.015188 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:18.015151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:18.109839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:18.109806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:18.838143 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:18.838105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:19.026955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:19.026919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:21.241841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:21.241813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:21.337506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:21.337469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:23.591194 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:23.591157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:23.655412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:23.655379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:24.020068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:24.020033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:24.127121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:24.127073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:24.400856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:24.400822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:24.505668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:24.505628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:29.868412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:29.868379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:29.968756 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:29.968724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:34.505485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:34.505453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:34.611336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:34.611302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:44.292974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.292939    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"]
Sep 10 11:22:44.315537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.315507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"]
Sep 10 11:22:44.315699 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.315610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.319132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.319110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4pw42\""
Sep 10 11:22:44.322144 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.322107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:44.407667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8nr\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-kube-api-access-8k8nr\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.408140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.408140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.407967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df9ea96-1641-4fba-8a1d-3000572c2b80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.509759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.509845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8nr\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-kube-api-access-8k8nr\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.509889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.509936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.509972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df9ea96-1641-4fba-8a1d-3000572c2b80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.510018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.510054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.510127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.510158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.510725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.510665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.511338 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.511287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.511793 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.511766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.512551 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.512529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df9ea96-1641-4fba-8a1d-3000572c2b80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.515457 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.515347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.515457 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.515392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df9ea96-1641-4fba-8a1d-3000572c2b80-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.516057 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.516041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.519600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.519578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.519962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.519940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8nr\" (UniqueName: \"kubernetes.io/projected/2df9ea96-1641-4fba-8a1d-3000572c2b80-kube-api-access-8k8nr\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4cj64\" (UID: \"2df9ea96-1641-4fba-8a1d-3000572c2b80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.535236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.535213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:44.626255 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.626226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:44.907789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.907766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"]
Sep 10 11:22:44.909543 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:22:44.909518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df9ea96_1641_4fba_8a1d_3000572c2b80.slice/crio-726c558592b8009effdf1de4c4cd58b9c870a6202f029dbb25b538bf87693732 WatchSource:0}: Error finding container 726c558592b8009effdf1de4c4cd58b9c870a6202f029dbb25b538bf87693732: Status 404 returned error can't find the container with id 726c558592b8009effdf1de4c4cd58b9c870a6202f029dbb25b538bf87693732
Sep 10 11:22:44.911865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.911833    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:22:44.911925 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.911900    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:22:44.911964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:44.911926    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:22:45.440886 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:45.440851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" event={"ID":"2df9ea96-1641-4fba-8a1d-3000572c2b80","Type":"ContainerStarted","Data":"cc0b0d4190ee4ec3b990f8dc09aeab394fde0126c56e9b33247a79357bd2dcbc"}
Sep 10 11:22:45.441387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:45.440916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" event={"ID":"2df9ea96-1641-4fba-8a1d-3000572c2b80","Type":"ContainerStarted","Data":"726c558592b8009effdf1de4c4cd58b9c870a6202f029dbb25b538bf87693732"}
Sep 10 11:22:45.473181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:45.473133    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podStartSLOduration=1.473119807 podStartE2EDuration="1.473119807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:22:45.471609393 +0000 UTC m=+483.546935056" watchObservedRunningTime="2026-09-10 11:22:45.473119807 +0000 UTC m=+483.548445470"
Sep 10 11:22:45.626858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:45.626817    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:45.631807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:45.631785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:46.445942 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:46.445910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:46.446770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:46.446741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:22:46.714496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:46.714412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:46.807325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:46.807297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:46.914986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:46.914957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:47.573189 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.573151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:47.576850 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.576827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:47.580096 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.580063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sqzbm\""
Sep 10 11:22:47.587834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.587809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:47.617376 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.617346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:47.638427 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.638398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bpk\" (UniqueName: \"kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk\") pod \"authorino-7bcfd8f9cd-dbn86\" (UID: \"95fe28e5-0b77-43ef-bca8-b07886a37223\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:47.711408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.711378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:47.739197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.739163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bpk\" (UniqueName: \"kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk\") pod \"authorino-7bcfd8f9cd-dbn86\" (UID: \"95fe28e5-0b77-43ef-bca8-b07886a37223\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:47.747900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.747874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bpk\" (UniqueName: \"kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk\") pod \"authorino-7bcfd8f9cd-dbn86\" (UID: \"95fe28e5-0b77-43ef-bca8-b07886a37223\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:47.886412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:47.886375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:48.158913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:48.158889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:48.161403 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:22:48.161379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95fe28e5_0b77_43ef_bca8_b07886a37223.slice/crio-3d917830a70746944e7ca7a65b333fb94a9a6a2f05dd9528d40e3e039b67582a WatchSource:0}: Error finding container 3d917830a70746944e7ca7a65b333fb94a9a6a2f05dd9528d40e3e039b67582a: Status 404 returned error can't find the container with id 3d917830a70746944e7ca7a65b333fb94a9a6a2f05dd9528d40e3e039b67582a
Sep 10 11:22:48.456611 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:48.456526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" event={"ID":"95fe28e5-0b77-43ef-bca8-b07886a37223","Type":"ContainerStarted","Data":"3d917830a70746944e7ca7a65b333fb94a9a6a2f05dd9528d40e3e039b67582a"}
Sep 10 11:22:50.467319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:50.467227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" event={"ID":"95fe28e5-0b77-43ef-bca8-b07886a37223","Type":"ContainerStarted","Data":"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"}
Sep 10 11:22:50.486059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:50.486005    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" podStartSLOduration=1.4719350310000001 podStartE2EDuration="3.485988974s" totalImagesPullingTime="2.014053943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:22:50.484415433 +0000 UTC m=+488.559741088" watchObservedRunningTime="2026-09-10 11:22:50.485988974 +0000 UTC m=+488.561314638"
Sep 10 11:22:50.934395 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:50.934361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:52.475917 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:52.475856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" podUID="95fe28e5-0b77-43ef-bca8-b07886a37223" containerName="authorino" containerID="cri-o://58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b" gracePeriod=30
Sep 10 11:22:52.713797 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:52.713769    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:52.782598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:52.782519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64bpk\" (UniqueName: \"kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk\") pod \"95fe28e5-0b77-43ef-bca8-b07886a37223\" (UID: \"95fe28e5-0b77-43ef-bca8-b07886a37223\") "
Sep 10 11:22:52.784779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:52.784749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk" (OuterVolumeSpecName: "kube-api-access-64bpk") pod "95fe28e5-0b77-43ef-bca8-b07886a37223" (UID: "95fe28e5-0b77-43ef-bca8-b07886a37223"). InnerVolumeSpecName "kube-api-access-64bpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:22:52.884013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:52.883977    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64bpk\" (UniqueName: \"kubernetes.io/projected/95fe28e5-0b77-43ef-bca8-b07886a37223-kube-api-access-64bpk\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:22:53.481607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.481575    2578 generic.go:356] "Generic (PLEG): container finished" podID="95fe28e5-0b77-43ef-bca8-b07886a37223" containerID="58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b" exitCode=0
Sep 10 11:22:53.482014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.481620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86"
Sep 10 11:22:53.482014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.481660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" event={"ID":"95fe28e5-0b77-43ef-bca8-b07886a37223","Type":"ContainerDied","Data":"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"}
Sep 10 11:22:53.482014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.481701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dbn86" event={"ID":"95fe28e5-0b77-43ef-bca8-b07886a37223","Type":"ContainerDied","Data":"3d917830a70746944e7ca7a65b333fb94a9a6a2f05dd9528d40e3e039b67582a"}
Sep 10 11:22:53.482014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.481718    2578 scope.go:117] "RemoveContainer" containerID="58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"
Sep 10 11:22:53.490209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.490189    2578 scope.go:117] "RemoveContainer" containerID="58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"
Sep 10 11:22:53.490478 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:22:53.490459    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b\": container with ID starting with 58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b not found: ID does not exist" containerID="58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"
Sep 10 11:22:53.490530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.490485    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b"} err="failed to get container status \"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b\": rpc error: code = NotFound desc = could not find container \"58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b\": container with ID starting with 58812d0923c6dc9c3c9ea90b8295b0ecc3fbe7594e9bd62f901b908ca89c6c3b not found: ID does not exist"
Sep 10 11:22:53.527390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.527358    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:53.534204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:53.534182    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dbn86"]
Sep 10 11:22:54.457059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:54.457015    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95fe28e5-0b77-43ef-bca8-b07886a37223" path="/var/lib/kubelet/pods/95fe28e5-0b77-43ef-bca8-b07886a37223/volumes"
Sep 10 11:22:56.648587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:56.648552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:57.627754 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:57.627721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:22:57.684982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:22:57.684946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:02.780862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:02.780829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:02.882158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:02.882125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:03.294182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:03.294153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:13.527442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:13.527406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:13.628032 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:13.627997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:14.086602 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:14.086557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:14.142006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:14.141977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:15.395156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.395129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:15.640608 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.640574    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:15.640880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.640869    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95fe28e5-0b77-43ef-bca8-b07886a37223" containerName="authorino"
Sep 10 11:23:15.640923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.640882    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95fe28e5-0b77-43ef-bca8-b07886a37223" containerName="authorino"
Sep 10 11:23:15.640957 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.640937    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95fe28e5-0b77-43ef-bca8-b07886a37223" containerName="authorino"
Sep 10 11:23:15.643790 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.643776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:15.648283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.648214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sqzbm\""
Sep 10 11:23:15.652165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.652141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:15.679522 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.679493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58n4\" (UniqueName: \"kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4\") pod \"authorino-557f876f69-nmw6c\" (UID: \"b2c5540c-07ee-4a21-89b8-a49716c6e1c9\") " pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:15.780458 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.780420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58n4\" (UniqueName: \"kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4\") pod \"authorino-557f876f69-nmw6c\" (UID: \"b2c5540c-07ee-4a21-89b8-a49716c6e1c9\") " pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:15.786863 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.786838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:15.789931 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.789906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58n4\" (UniqueName: \"kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4\") pod \"authorino-557f876f69-nmw6c\" (UID: \"b2c5540c-07ee-4a21-89b8-a49716c6e1c9\") " pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:15.869219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.869181    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:15.869498 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.869485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:15.902374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.902295    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:23:15.906195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.906172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:15.909163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.909132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 11:23:15.916370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.916344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:23:15.982486 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.982450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98f28\" (UniqueName: \"kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:15.982648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:15.982540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.083448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.083414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98f28\" (UniqueName: \"kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.083565 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.083467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.086037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.086014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.091851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.091826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98f28\" (UniqueName: \"kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28\") pod \"authorino-555684b565-smlg4\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") " pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.145827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.145804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:16.147678 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:23:16.147648    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c5540c_07ee_4a21_89b8_a49716c6e1c9.slice/crio-fdde7c44d57d88fde782805b873c83b879246113146040affd076e7ccab32934 WatchSource:0}: Error finding container fdde7c44d57d88fde782805b873c83b879246113146040affd076e7ccab32934: Status 404 returned error can't find the container with id fdde7c44d57d88fde782805b873c83b879246113146040affd076e7ccab32934
Sep 10 11:23:16.217388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.217322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:23:16.489561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.489490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:23:16.492856 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:23:16.492828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be20ac7_d1f0_4ea5_af7a_e3eb6e8fd61f.slice/crio-3e6fb1acf1ae7db645528d1053f6cbb7e738e3f2cdf2adba7d6e7562db59bc73 WatchSource:0}: Error finding container 3e6fb1acf1ae7db645528d1053f6cbb7e738e3f2cdf2adba7d6e7562db59bc73: Status 404 returned error can't find the container with id 3e6fb1acf1ae7db645528d1053f6cbb7e738e3f2cdf2adba7d6e7562db59bc73
Sep 10 11:23:16.595168 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.595130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555684b565-smlg4" event={"ID":"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f","Type":"ContainerStarted","Data":"3e6fb1acf1ae7db645528d1053f6cbb7e738e3f2cdf2adba7d6e7562db59bc73"}
Sep 10 11:23:16.596267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:16.596246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nmw6c" event={"ID":"b2c5540c-07ee-4a21-89b8-a49716c6e1c9","Type":"ContainerStarted","Data":"fdde7c44d57d88fde782805b873c83b879246113146040affd076e7ccab32934"}
Sep 10 11:23:17.601993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.601947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555684b565-smlg4" event={"ID":"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f","Type":"ContainerStarted","Data":"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"}
Sep 10 11:23:17.603628 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.603598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nmw6c" event={"ID":"b2c5540c-07ee-4a21-89b8-a49716c6e1c9","Type":"ContainerStarted","Data":"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"}
Sep 10 11:23:17.603766 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.603630    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-nmw6c" podUID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" containerName="authorino" containerID="cri-o://427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd" gracePeriod=30
Sep 10 11:23:17.641594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.641541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-555684b565-smlg4" podStartSLOduration=2.079885881 podStartE2EDuration="2.641526576s" totalImagesPullingTime="561.640695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:23:17.636968018 +0000 UTC m=+515.712293682" watchObservedRunningTime="2026-09-10 11:23:17.641526576 +0000 UTC m=+515.716852239"
Sep 10 11:23:17.849798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.849772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:17.898509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.898477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w58n4\" (UniqueName: \"kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4\") pod \"b2c5540c-07ee-4a21-89b8-a49716c6e1c9\" (UID: \"b2c5540c-07ee-4a21-89b8-a49716c6e1c9\") "
Sep 10 11:23:17.900607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.900578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4" (OuterVolumeSpecName: "kube-api-access-w58n4") pod "b2c5540c-07ee-4a21-89b8-a49716c6e1c9" (UID: "b2c5540c-07ee-4a21-89b8-a49716c6e1c9"). InnerVolumeSpecName "kube-api-access-w58n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:23:17.999025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:17.998988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58n4\" (UniqueName: \"kubernetes.io/projected/b2c5540c-07ee-4a21-89b8-a49716c6e1c9-kube-api-access-w58n4\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:23:18.609853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.609817    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" containerID="427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd" exitCode=0
Sep 10 11:23:18.609853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.609850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nmw6c"
Sep 10 11:23:18.610329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.609883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nmw6c" event={"ID":"b2c5540c-07ee-4a21-89b8-a49716c6e1c9","Type":"ContainerDied","Data":"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"}
Sep 10 11:23:18.610329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.609920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nmw6c" event={"ID":"b2c5540c-07ee-4a21-89b8-a49716c6e1c9","Type":"ContainerDied","Data":"fdde7c44d57d88fde782805b873c83b879246113146040affd076e7ccab32934"}
Sep 10 11:23:18.610329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.609935    2578 scope.go:117] "RemoveContainer" containerID="427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"
Sep 10 11:23:18.618147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.618131    2578 scope.go:117] "RemoveContainer" containerID="427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"
Sep 10 11:23:18.618398 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:23:18.618370    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd\": container with ID starting with 427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd not found: ID does not exist" containerID="427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"
Sep 10 11:23:18.618450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.618398    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd"} err="failed to get container status \"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd\": rpc error: code = NotFound desc = could not find container \"427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd\": container with ID starting with 427f7712d8034e4024b24b34acb1002aa7b35255f1d4bf1009bdc29736b01afd not found: ID does not exist"
Sep 10 11:23:18.627816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.627794    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:18.631888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:18.631868    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-nmw6c"]
Sep 10 11:23:20.455117 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:20.455068    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" path="/var/lib/kubelet/pods/b2c5540c-07ee-4a21-89b8-a49716c6e1c9/volumes"
Sep 10 11:23:23.248111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:23.248061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:23.380767 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:23.380731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:23.455006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:23.454971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:23.552529 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:23.552450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:23.693296 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:23.693264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:26.987260 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:26.987232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:27.189245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:27.189201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:58.763527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:58.763488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:23:58.861394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:23:58.861358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.166460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.166427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.267043 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.267006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.408028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.408000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.507823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.507734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.748443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.748413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:00.853297 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:00.853211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:06.722826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:06.722792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:10.370812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.370774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:10.471433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.471350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:10.851301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.851201    2578 generic.go:356] "Generic (PLEG): container finished" podID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerID="cc0b0d4190ee4ec3b990f8dc09aeab394fde0126c56e9b33247a79357bd2dcbc" exitCode=0
Sep 10 11:24:10.851301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.851245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" event={"ID":"2df9ea96-1641-4fba-8a1d-3000572c2b80","Type":"ContainerDied","Data":"cc0b0d4190ee4ec3b990f8dc09aeab394fde0126c56e9b33247a79357bd2dcbc"}
Sep 10 11:24:10.851662 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.851644    2578 scope.go:117] "RemoveContainer" containerID="cc0b0d4190ee4ec3b990f8dc09aeab394fde0126c56e9b33247a79357bd2dcbc"
Sep 10 11:24:10.852641 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.852611    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:24:10.852698 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.852688    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:24:10.852731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:10.852721    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:24:11.858459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:11.858420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" event={"ID":"2df9ea96-1641-4fba-8a1d-3000572c2b80","Type":"ContainerStarted","Data":"8102f31752d2750a8d5a68980d391479a7234d5e99dfbabec4f0680217697283"}
Sep 10 11:24:12.627162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:12.627125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:24:12.628255 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:12.628227    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4cj64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 10 11:24:12.628373 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:12.628282    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podUID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 10 11:24:13.627536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:13.627498    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4cj64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 10 11:24:13.627909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:13.627559    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podUID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 10 11:24:14.627250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:14.627214    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4cj64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 10 11:24:14.627447 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:14.627273    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podUID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 10 11:24:15.627128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:15.627064    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4cj64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 10 11:24:15.627512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:15.627179    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podUID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 10 11:24:16.446492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:16.446453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:24:16.627675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:16.627629    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4cj64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 10 11:24:16.628056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:16.627697    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64" podUID="2df9ea96-1641-4fba-8a1d-3000572c2b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 10 11:24:17.262516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.262478    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"]
Sep 10 11:24:17.262847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.262833    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" containerName="authorino"
Sep 10 11:24:17.262896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.262850    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" containerName="authorino"
Sep 10 11:24:17.262930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.262921    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c5540c-07ee-4a21-89b8-a49716c6e1c9" containerName="authorino"
Sep 10 11:24:17.265962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.265945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.269296 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.269276    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 11:24:17.269519 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.269503    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 11:24:17.269603 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.269516    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 11:24:17.271124 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.271106    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5ft7p\""
Sep 10 11:24:17.279938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.279917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"]
Sep 10 11:24:17.304859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.304818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:17.374021 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.373978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.374223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.374025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.374223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.374124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.374223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.374160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e77b37-d395-4e21-ad01-c796c8cd4282-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.374386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.374234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfd2\" (UniqueName: \"kubernetes.io/projected/89e77b37-d395-4e21-ad01-c796c8cd4282-kube-api-access-9sfd2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.374386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.374293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e77b37-d395-4e21-ad01-c796c8cd4282-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfd2\" (UniqueName: \"kubernetes.io/projected/89e77b37-d395-4e21-ad01-c796c8cd4282-kube-api-access-9sfd2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.475750 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.475687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.476073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.476040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.476183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.476146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.476183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.476179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.478051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.478032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e77b37-d395-4e21-ad01-c796c8cd4282-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.478206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.478187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e77b37-d395-4e21-ad01-c796c8cd4282-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.485257 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.485224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfd2\" (UniqueName: \"kubernetes.io/projected/89e77b37-d395-4e21-ad01-c796c8cd4282-kube-api-access-9sfd2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl\" (UID: \"89e77b37-d395-4e21-ad01-c796c8cd4282\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.575806 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.575703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:17.632076 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.632045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:24:17.634078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.634055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4cj64"
Sep 10 11:24:17.862474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.862451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"]
Sep 10 11:24:17.865119 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:24:17.865069    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e77b37_d395_4e21_ad01_c796c8cd4282.slice/crio-fa1078dcd48910fc5f2fc403dad22856daf39d799ad1406811db736c80f4b855 WatchSource:0}: Error finding container fa1078dcd48910fc5f2fc403dad22856daf39d799ad1406811db736c80f4b855: Status 404 returned error can't find the container with id fa1078dcd48910fc5f2fc403dad22856daf39d799ad1406811db736c80f4b855
Sep 10 11:24:17.890035 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:17.889979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl" event={"ID":"89e77b37-d395-4e21-ad01-c796c8cd4282","Type":"ContainerStarted","Data":"fa1078dcd48910fc5f2fc403dad22856daf39d799ad1406811db736c80f4b855"}
Sep 10 11:24:19.902399 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:19.902364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl" event={"ID":"89e77b37-d395-4e21-ad01-c796c8cd4282","Type":"ContainerStarted","Data":"c3cf9b4356c599fe44dd0ca952df43f7e35120aa185786b97efdd871d30cfa77"}
Sep 10 11:24:19.929791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:19.929739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl" podStartSLOduration=1.7768414689999998 podStartE2EDuration="2.929724428s" totalImagesPullingTime="1.152882959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:24:19.928360114 +0000 UTC m=+578.003685780" watchObservedRunningTime="2026-09-10 11:24:19.929724428 +0000 UTC m=+578.005050092"
Sep 10 11:24:27.219860 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:27.219820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:27.576319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:27.576231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:27.576472 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:27.576352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:27.588890 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:27.588863    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:27.953968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:27.953937    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl"
Sep 10 11:24:31.268293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:31.268263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:42.387375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:42.387345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:24:42.388011 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:42.387988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:24:45.241369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.241329    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"]
Sep 10 11:24:45.245024 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.245003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.248066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.248043    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 11:24:45.254977 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.254954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"]
Sep 10 11:24:45.276343 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.276314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:45.311610 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.311755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af934cc-954d-473d-85f5-755b8c54b553-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.311755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.311755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.311892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5fm\" (UniqueName: \"kubernetes.io/projected/3af934cc-954d-473d-85f5-755b8c54b553-kube-api-access-6b5fm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.311892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.311838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413272 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5fm\" (UniqueName: \"kubernetes.io/projected/3af934cc-954d-473d-85f5-755b8c54b553-kube-api-access-6b5fm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af934cc-954d-473d-85f5-755b8c54b553-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413804 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.413842 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.413811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.415784 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.415757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af934cc-954d-473d-85f5-755b8c54b553-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.416045 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.416026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af934cc-954d-473d-85f5-755b8c54b553-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.422196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.422170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5fm\" (UniqueName: \"kubernetes.io/projected/3af934cc-954d-473d-85f5-755b8c54b553-kube-api-access-6b5fm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z\" (UID: \"3af934cc-954d-473d-85f5-755b8c54b553\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.556804 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.556720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:45.836219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.836145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"]
Sep 10 11:24:45.839281 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:24:45.839250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af934cc_954d_473d_85f5_755b8c54b553.slice/crio-5c24adaa052e67d89f20cd3eb11cbdb46b19df9ef9babd044d843dd17af31711 WatchSource:0}: Error finding container 5c24adaa052e67d89f20cd3eb11cbdb46b19df9ef9babd044d843dd17af31711: Status 404 returned error can't find the container with id 5c24adaa052e67d89f20cd3eb11cbdb46b19df9ef9babd044d843dd17af31711
Sep 10 11:24:45.840789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:45.840769    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:24:46.024530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:46.024484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z" event={"ID":"3af934cc-954d-473d-85f5-755b8c54b553","Type":"ContainerStarted","Data":"5c24adaa052e67d89f20cd3eb11cbdb46b19df9ef9babd044d843dd17af31711"}
Sep 10 11:24:47.031259 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:47.031227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z" event={"ID":"3af934cc-954d-473d-85f5-755b8c54b553","Type":"ContainerStarted","Data":"35b1f8371c1a3c6dd8b196ac47d63607eeea8dbedf4ecf2fd5320cf5f1db2d90"}
Sep 10 11:24:55.233271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:55.233239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:24:55.557380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:55.557294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:55.557607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:55.557580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:55.570242 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:55.570215    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:24:55.590626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:55.590576    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z" podStartSLOduration=10.400292402 podStartE2EDuration="10.590560146s" totalImagesPullingTime="190.267744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:24:47.061024805 +0000 UTC m=+605.136350470" watchObservedRunningTime="2026-09-10 11:24:55.590560146 +0000 UTC m=+613.665885809"
Sep 10 11:24:56.083896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:24:56.083869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z"
Sep 10 11:25:04.467451 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:25:04.467378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:25:14.078579 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:25:14.078543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:25:16.271676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:25:16.271641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:24.630579 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.630543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-646d785fb4-fb5pd"]
Sep 10 11:26:24.633859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.633833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.652649 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.652617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646d785fb4-fb5pd"]
Sep 10 11:26:24.681469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.681440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4eaeef07-98e6-4bad-901e-f111fc1d9aad-tls-cert\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.681619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.681488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgp8\" (UniqueName: \"kubernetes.io/projected/4eaeef07-98e6-4bad-901e-f111fc1d9aad-kube-api-access-bqgp8\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.782444 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.782405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4eaeef07-98e6-4bad-901e-f111fc1d9aad-tls-cert\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.782610 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.782460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgp8\" (UniqueName: \"kubernetes.io/projected/4eaeef07-98e6-4bad-901e-f111fc1d9aad-kube-api-access-bqgp8\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.785101 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.785052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4eaeef07-98e6-4bad-901e-f111fc1d9aad-tls-cert\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.792929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.792906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgp8\" (UniqueName: \"kubernetes.io/projected/4eaeef07-98e6-4bad-901e-f111fc1d9aad-kube-api-access-bqgp8\") pod \"authorino-646d785fb4-fb5pd\" (UID: \"4eaeef07-98e6-4bad-901e-f111fc1d9aad\") " pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:24.943303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:24.943217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646d785fb4-fb5pd"
Sep 10 11:26:25.214215 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:25.214191    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646d785fb4-fb5pd"]
Sep 10 11:26:25.216272 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:26:25.216243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eaeef07_98e6_4bad_901e_f111fc1d9aad.slice/crio-11a9e22d80bf4b90325e876cfe8aeed0862bc0d44733adceeac65656ef8a5bff WatchSource:0}: Error finding container 11a9e22d80bf4b90325e876cfe8aeed0862bc0d44733adceeac65656ef8a5bff: Status 404 returned error can't find the container with id 11a9e22d80bf4b90325e876cfe8aeed0862bc0d44733adceeac65656ef8a5bff
Sep 10 11:26:25.486308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:25.486223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646d785fb4-fb5pd" event={"ID":"4eaeef07-98e6-4bad-901e-f111fc1d9aad","Type":"ContainerStarted","Data":"11a9e22d80bf4b90325e876cfe8aeed0862bc0d44733adceeac65656ef8a5bff"}
Sep 10 11:26:26.502452 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.502411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646d785fb4-fb5pd" event={"ID":"4eaeef07-98e6-4bad-901e-f111fc1d9aad","Type":"ContainerStarted","Data":"11ee437a82e75b61a4944ff066f571f79cc13b378c7c128df7f55e22bcdf8ad9"}
Sep 10 11:26:26.529323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.529266    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-646d785fb4-fb5pd" podStartSLOduration=1.7796973139999999 podStartE2EDuration="2.52925108s" totalImagesPullingTime="749.553766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:26:26.52705295 +0000 UTC m=+704.602378615" watchObservedRunningTime="2026-09-10 11:26:26.52925108 +0000 UTC m=+704.604576743"
Sep 10 11:26:26.579926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.579881    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:26:26.580251 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.580216    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-555684b565-smlg4" podUID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" containerName="authorino" containerID="cri-o://9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f" gracePeriod=30
Sep 10 11:26:26.838371 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.838346    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:26:26.905307 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.905260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98f28\" (UniqueName: \"kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28\") pod \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") "
Sep 10 11:26:26.905493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.905361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert\") pod \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\" (UID: \"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f\") "
Sep 10 11:26:26.908328 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.908292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28" (OuterVolumeSpecName: "kube-api-access-98f28") pod "8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" (UID: "8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f"). InnerVolumeSpecName "kube-api-access-98f28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:26:26.916855 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:26.916826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" (UID: "8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:26:27.006627 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.006590    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-tls-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:26:27.006627 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.006622    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98f28\" (UniqueName: \"kubernetes.io/projected/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f-kube-api-access-98f28\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:26:27.287731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.287667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:27.509434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.509397    2578 generic.go:356] "Generic (PLEG): container finished" podID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" containerID="9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f" exitCode=0
Sep 10 11:26:27.509883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.509446    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555684b565-smlg4"
Sep 10 11:26:27.509883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.509482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555684b565-smlg4" event={"ID":"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f","Type":"ContainerDied","Data":"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"}
Sep 10 11:26:27.509883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.509519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555684b565-smlg4" event={"ID":"8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f","Type":"ContainerDied","Data":"3e6fb1acf1ae7db645528d1053f6cbb7e738e3f2cdf2adba7d6e7562db59bc73"}
Sep 10 11:26:27.509883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.509532    2578 scope.go:117] "RemoveContainer" containerID="9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"
Sep 10 11:26:27.519823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.519802    2578 scope.go:117] "RemoveContainer" containerID="9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"
Sep 10 11:26:27.520148 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:26:27.520113    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f\": container with ID starting with 9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f not found: ID does not exist" containerID="9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"
Sep 10 11:26:27.520299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.520148    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f"} err="failed to get container status \"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f\": rpc error: code = NotFound desc = could not find container \"9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f\": container with ID starting with 9742f66f6cf65359759d6588690d47abc4bbe413f3d76820f0c1c78e3ce4320f not found: ID does not exist"
Sep 10 11:26:27.543932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.543850    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:26:27.547333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:27.547301    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-555684b565-smlg4"]
Sep 10 11:26:28.455306 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:28.455271    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" path="/var/lib/kubelet/pods/8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f/volumes"
Sep 10 11:26:29.160600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:29.160564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:47.163450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.163415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:47.260296 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.260251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:47.472840 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.472755    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:26:47.473184 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.473167    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" containerName="authorino"
Sep 10 11:26:47.473276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.473186    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" containerName="authorino"
Sep 10 11:26:47.473276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.473272    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be20ac7-d1f0-4ea5-af7a-e3eb6e8fd61f" containerName="authorino"
Sep 10 11:26:47.477383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.477363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.480300 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.480283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8355df-openshift-default-dockercfg-8r9sm\""
Sep 10 11:26:47.493614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.493590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:26:47.495408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.495384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:47.592702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.592661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.592953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.592926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593074 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.592963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593074 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.593389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.593297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmbq\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.671618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.671580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:47.694161 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmbq\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.694672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.694468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.695276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.695217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.695401 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.695377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.695589 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.695566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.695676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.695626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.695676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.695636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.697138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.697118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.697224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.697153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.704023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.704000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.704246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.704221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmbq\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq\") pod \"e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") " pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:47.789133 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:47.789009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:48.073557 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.073457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:26:48.077338 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:26:48.077307    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b3aa34_1b86_4bdc_a6cf_06362bcb772f.slice/crio-8f6114b6fde4aefb4bf8e793fd6ef161f71137763481a80614c2a1200eb0b676 WatchSource:0}: Error finding container 8f6114b6fde4aefb4bf8e793fd6ef161f71137763481a80614c2a1200eb0b676: Status 404 returned error can't find the container with id 8f6114b6fde4aefb4bf8e793fd6ef161f71137763481a80614c2a1200eb0b676
Sep 10 11:26:48.079864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.079828    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:48.079963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.079912    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:48.079963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.079957    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:48.607108 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.607065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerStarted","Data":"b7b163e4acac6bb18df79ead40d24b062a2d0e38695a260a4bc1545ac90a9850"}
Sep 10 11:26:48.607505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.607139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerStarted","Data":"8f6114b6fde4aefb4bf8e793fd6ef161f71137763481a80614c2a1200eb0b676"}
Sep 10 11:26:48.635736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.635682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podStartSLOduration=1.635663032 podStartE2EDuration="1.635663032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:26:48.633398885 +0000 UTC m=+726.708724548" watchObservedRunningTime="2026-09-10 11:26:48.635663032 +0000 UTC m=+726.710988696"
Sep 10 11:26:48.680201 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.680169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:48.789149 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.789112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:48.789796 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:48.789763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:49.218881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:49.218844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:49.795550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:49.795518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:50.617381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:50.617356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:50.618335 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:50.618313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:26:50.781642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:50.781603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:50.878936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:50.878854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:51.993241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:51.993205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:52.162299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.162261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:52.264496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.264413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:52.695668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.695637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:52.800409 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.800373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:52.809792 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.809762    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:26:52.813452 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.813433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:52.818037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.818016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8355df\""
Sep 10 11:26:52.826613 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.826591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:26:52.941558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.941522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w25l\" (UniqueName: \"kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:52.941756 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:52.941637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.042301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.042207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w25l\" (UniqueName: \"kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.042301 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.042293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.042974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.042954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.053183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.053158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w25l\" (UniqueName: \"kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l\") pod \"payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.123505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.123472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:53.398770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.398682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:26:53.401704 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:26:53.401678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c462f9f_1de1_45ab_9165_2b79d9b2d52a.slice/crio-d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7 WatchSource:0}: Error finding container d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7: Status 404 returned error can't find the container with id d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7
Sep 10 11:26:53.632920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:53.632882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" event={"ID":"2c462f9f-1de1-45ab-9165-2b79d9b2d52a","Type":"ContainerStarted","Data":"d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7"}
Sep 10 11:26:55.617370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:55.617342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:55.711672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:55.711639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:55.863838 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:55.863800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:56.135991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.135954    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:26:56.139460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.139440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.142642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.142621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aff377da-openshift-default-dockercfg-9tqhh\""
Sep 10 11:26:56.152594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.152571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:26:56.169661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxbs\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.169858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.170116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.169869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.174209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.174178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:56.270671 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270671 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxbs\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.270902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.270907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.271179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271368 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.271343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.271422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.271426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.271539 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.271449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.273342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.273320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.273547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.273532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.279533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.279511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxbs\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.279766 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.279746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token\") pod \"e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") " pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.449884 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.449819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:56.472798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.472768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:56.651523 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.651474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" event={"ID":"2c462f9f-1de1-45ab-9165-2b79d9b2d52a","Type":"ContainerStarted","Data":"d468078a8531746b1f674abe446880afbc45b1ee64cb8e6f2052ef113c167aec"}
Sep 10 11:26:56.651927 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.651625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:26:56.671652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.671496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" podStartSLOduration=2.073047162 podStartE2EDuration="4.671474984s" totalImagesPullingTime="2.598427822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:26:56.669901184 +0000 UTC m=+734.745226849" watchObservedRunningTime="2026-09-10 11:26:56.671474984 +0000 UTC m=+734.746800695"
Sep 10 11:26:56.754554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.754529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:26:56.755246 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:26:56.755223    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c38e4f4_c726_4ac4_bf94_ed8fd83c1567.slice/crio-f03ae3340df7e814c06165b47e27506b77d93d93443d6bf58a6917676ba89c98 WatchSource:0}: Error finding container f03ae3340df7e814c06165b47e27506b77d93d93443d6bf58a6917676ba89c98: Status 404 returned error can't find the container with id f03ae3340df7e814c06165b47e27506b77d93d93443d6bf58a6917676ba89c98
Sep 10 11:26:56.757462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.757430    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:56.757530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.757496    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:56.757530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:56.757523    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:57.657561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:57.657515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerStarted","Data":"2690906dd1e6efdc12762ff5911fd71a3cc5c2032e7c6e15ec2072226971dd40"}
Sep 10 11:26:57.657981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:57.657613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerStarted","Data":"f03ae3340df7e814c06165b47e27506b77d93d93443d6bf58a6917676ba89c98"}
Sep 10 11:26:57.685506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:57.685452    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podStartSLOduration=1.6854313269999999 podStartE2EDuration="1.685431327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:26:57.68293438 +0000 UTC m=+735.758260046" watchObservedRunningTime="2026-09-10 11:26:57.685431327 +0000 UTC m=+735.760756992"
Sep 10 11:26:57.781275 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:57.781242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:58.092160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.086351    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:26:58.107112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.100370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.107112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.105565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:26:58.107112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.105894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8de26b-openshift-default-dockercfg-5s2tw\""
Sep 10 11:26:58.188999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.188954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189290 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zx97\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189778 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.189892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.189879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.190018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.190005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.190146 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.190130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.203111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.202593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:58.291015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.290975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.291345 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.291477 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.291599 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.291734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.291895 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.292005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.291990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.292224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.292207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.292362 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.292347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx97\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.293378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.293358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.294437 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.294417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.294949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.294869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.295281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.295167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.295563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.295529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.297512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.297491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.298458 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.298437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.307107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.305427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.307107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.305976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx97\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97\") pod \"e2e-autores-8de26b-openshift-default-774f69876-tkckj\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") " pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.429527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.429492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:26:58.457356 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.457325    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:58.457540 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.457444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:58.513807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.511762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:58.662333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.662303    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:58.663459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.663438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:26:58.746049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.745967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:26:58.749633 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:26:58.749605    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e65cb15_18fc_47cb_80ba_9c132702a099.slice/crio-97ade14a73a28b83fef449ed94b7b342e75491b2f89f51160999fa248a578e68 WatchSource:0}: Error finding container 97ade14a73a28b83fef449ed94b7b342e75491b2f89f51160999fa248a578e68: Status 404 returned error can't find the container with id 97ade14a73a28b83fef449ed94b7b342e75491b2f89f51160999fa248a578e68
Sep 10 11:26:58.752476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.752443    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:58.752548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.752509    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:58.752548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:58.752539    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:26:59.179945 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:59.179913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:59.279718 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:59.279688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:26:59.668015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:59.667978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerStarted","Data":"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"}
Sep 10 11:26:59.668414 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:59.668039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerStarted","Data":"97ade14a73a28b83fef449ed94b7b342e75491b2f89f51160999fa248a578e68"}
Sep 10 11:26:59.696379 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:26:59.696316    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podStartSLOduration=1.696294817 podStartE2EDuration="1.696294817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:26:59.693294937 +0000 UTC m=+737.768620613" watchObservedRunningTime="2026-09-10 11:26:59.696294817 +0000 UTC m=+737.771620483"
Sep 10 11:27:00.357755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.357722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:00.430375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.430336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:00.435272 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.435248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:00.464871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.464815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:00.565513 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.565478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:00.672693 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.672599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:00.673670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:00.673653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:01.089048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.088946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:01.302583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.302546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:01.417988 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.417955    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:01.421419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.421404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.424600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.424578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aff377da-dockercfg-tb59h\""
Sep 10 11:27:01.424867 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.424852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aff377da\""
Sep 10 11:27:01.430500 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.430480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.430593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.430516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd8d\" (UniqueName: \"kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.442626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.442602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:01.531040 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.531000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.531219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.531051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd8d\" (UniqueName: \"kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.531763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.531742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.541495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.541460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd8d\" (UniqueName: \"kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d\") pod \"payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.711639 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.711545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:01.731543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.731507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:01.821074 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:01.820921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:02.014025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:02.013998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:02.016285 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:27:02.016259    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1052de84_b4e4_4a4c_8ac4_65650e0ec9bd.slice/crio-3d11fbdb7d30a32e2d1cf88c7a9fd5d649b28e94342619c8eb23ebdf84e2a992 WatchSource:0}: Error finding container 3d11fbdb7d30a32e2d1cf88c7a9fd5d649b28e94342619c8eb23ebdf84e2a992: Status 404 returned error can't find the container with id 3d11fbdb7d30a32e2d1cf88c7a9fd5d649b28e94342619c8eb23ebdf84e2a992
Sep 10 11:27:02.684123 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:02.684047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" event={"ID":"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd","Type":"ContainerStarted","Data":"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"}
Sep 10 11:27:02.684427 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:02.684145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" event={"ID":"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd","Type":"ContainerStarted","Data":"3d11fbdb7d30a32e2d1cf88c7a9fd5d649b28e94342619c8eb23ebdf84e2a992"}
Sep 10 11:27:02.684427 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:02.684227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:02.711353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:02.711295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" podStartSLOduration=1.7112734710000002 podStartE2EDuration="1.711273471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:27:02.706069234 +0000 UTC m=+740.781394899" watchObservedRunningTime="2026-09-10 11:27:02.711273471 +0000 UTC m=+740.786599136"
Sep 10 11:27:03.025979 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:03.025892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:03.130171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:03.130140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:03.275796 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:03.275768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:03.570340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:03.570310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:03.671331 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:03.671294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:06.112591 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.112550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:06.268365 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.268333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:06.390913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.390823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:06.491928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.491883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:06.659328 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.659246    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 11:27:06.659474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:06.659327    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 11:27:07.659446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.659409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:27:07.727521 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.727482    2578 generic.go:356] "Generic (PLEG): container finished" podID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerID="b7b163e4acac6bb18df79ead40d24b062a2d0e38695a260a4bc1545ac90a9850" exitCode=0
Sep 10 11:27:07.727683 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.727568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerDied","Data":"b7b163e4acac6bb18df79ead40d24b062a2d0e38695a260a4bc1545ac90a9850"}
Sep 10 11:27:07.728135 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.728103    2578 scope.go:117] "RemoveContainer" containerID="b7b163e4acac6bb18df79ead40d24b062a2d0e38695a260a4bc1545ac90a9850"
Sep 10 11:27:07.729339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.729300    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:07.729446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.729410    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:07.729511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:07.729470    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:08.355160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.355124    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:27:08.358702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.358684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.361949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.361879    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-27987e85-kserve-self-signed-certs\""
Sep 10 11:27:08.379235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.379177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:27:08.412147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.406004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:08.413345 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.413306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.413585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.413569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.413738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.413725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.413873 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.413860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.414029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.414015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.414269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.414250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895g6\" (UniqueName: \"kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.512109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.511157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895g6\" (UniqueName: \"kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.516438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.517340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.517845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.523111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.518078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.527107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.527011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.535167 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.534270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.536619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.536575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-895g6\" (UniqueName: \"kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6\") pod \"e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") " pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.617113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.615812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:08.668454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.668422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:08.716669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.716603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:08.734433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.734399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerStarted","Data":"5310e05d07bcdb108b0ff27bb0bc409fb964bfd893e9a9b0612868ca9a5d2087"}
Sep 10 11:27:08.801113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.797816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:27:08.801113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.798438    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:08.801113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.798496    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:08.975638 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:08.975611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:27:08.977121 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:27:08.977066    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod442af6f2_d7bf_4e38_9fbb_7ae6830965ab.slice/crio-afa8457566e45567c564e29a26ae59558c0613acaf3a26f4027a4220d0d231fc WatchSource:0}: Error finding container afa8457566e45567c564e29a26ae59558c0613acaf3a26f4027a4220d0d231fc: Status 404 returned error can't find the container with id afa8457566e45567c564e29a26ae59558c0613acaf3a26f4027a4220d0d231fc
Sep 10 11:27:09.246667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.246582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:09.741743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.741654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" event={"ID":"442af6f2-d7bf-4e38-9fbb-7ae6830965ab","Type":"ContainerStarted","Data":"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"}
Sep 10 11:27:09.741743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.741710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" event={"ID":"442af6f2-d7bf-4e38-9fbb-7ae6830965ab","Type":"ContainerStarted","Data":"afa8457566e45567c564e29a26ae59558c0613acaf3a26f4027a4220d0d231fc"}
Sep 10 11:27:09.771185 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.771120    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" podStartSLOduration=1.771099384 podStartE2EDuration="1.771099384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:27:09.767925051 +0000 UTC m=+747.843250716" watchObservedRunningTime="2026-09-10 11:27:09.771099384 +0000 UTC m=+747.846425042"
Sep 10 11:27:09.803100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.802233    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:09.803100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:09.802297    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:10.790144 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:10.789668    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:10.790144 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:10.789731    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:11.038224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:11.038175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:11.790490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:11.790443    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:11.791003 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:11.790514    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:12.789502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:12.789454    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:12.789715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:12.789528    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:13.663821 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.663783    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:13.664246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.663851    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:13.701039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.700271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:13.769666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.769563    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerID="2690906dd1e6efdc12762ff5911fd71a3cc5c2032e7c6e15ec2072226971dd40" exitCode=0
Sep 10 11:27:13.769666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.769636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerDied","Data":"2690906dd1e6efdc12762ff5911fd71a3cc5c2032e7c6e15ec2072226971dd40"}
Sep 10 11:27:13.770044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.770031    2578 scope.go:117] "RemoveContainer" containerID="2690906dd1e6efdc12762ff5911fd71a3cc5c2032e7c6e15ec2072226971dd40"
Sep 10 11:27:13.771283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.771247    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:13.771387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.771366    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:13.771443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.771423    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:13.799120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.792508    2578 patch_prober.go:28] interesting pod/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 11:27:13.799120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:13.792583    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 11:27:14.369006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:14.368972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:14.777039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:14.776951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerStarted","Data":"2f45f41fef554b39a11eef0eb82aa60341899fe8f6b90d9a6873486a80313a29"}
Sep 10 11:27:14.798411 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:14.798374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:27:14.799120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:14.799058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:27:14.801662 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:14.801636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:27:15.232393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.232334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:15.450671 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.450629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:15.450861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.450738    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:15.450861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.450774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:15.654875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.654836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:15.755043 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.754978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:15.890754 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:15.890723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:16.055051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.054960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:16.129631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.129592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:16.357389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.357353    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:27:16.361189 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.361166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.364911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.364892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4c9990-openshift-default-dockercfg-pgf7f\""
Sep 10 11:27:16.382795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.382771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:27:16.419600 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.419428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:16.450318 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.450288    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:16.450567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.450544    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:16.520110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.519755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctr6\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.519893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.520487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.520383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.621829 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.621829 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.621829 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctr6\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.621997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.622022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622546 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.622520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.622829 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.622790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.623986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.623964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.626037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.624767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.626244 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.625457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.626378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.625923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.627010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.626988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.636317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.636267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.639136 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.639111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctr6\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6\") pod \"e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") " pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.671220 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.671183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:16.742140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:16.736592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:17.018171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.017842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:27:17.033005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.032346    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:17.033005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.032444    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:17.033005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.032490    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:17.450761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.450336    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:17.450761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.450402    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:17.813195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.813096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerStarted","Data":"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"}
Sep 10 11:27:17.813460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.813431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerStarted","Data":"a0cad19258b60e1f0483d83da7a7d626cf6f7a1c004060cf9cb0d495931689e4"}
Sep 10 11:27:17.842214 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:17.842150    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podStartSLOduration=1.842125888 podStartE2EDuration="1.842125888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:27:17.838916929 +0000 UTC m=+755.914242586" watchObservedRunningTime="2026-09-10 11:27:17.842125888 +0000 UTC m=+755.917451559"
Sep 10 11:27:18.450381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.450340    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:18.450870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.450422    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:18.669444 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.669407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:18.669645 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.669470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:18.671400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.671375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:18.676209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.676178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:18.696606 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.696574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:18.818948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.818863    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:18.819954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.819932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:18.834589 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:18.834563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:27:19.451237 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:19.451196    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:19.451705 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:19.451267    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:19.889685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:19.889579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:20.009687 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:20.009555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:20.450322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:20.450283    2578 patch_prober.go:28] interesting pod/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 11:27:20.450476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:20.450355    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 11:27:21.254154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.254123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:21.454684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.454654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:21.454969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.454950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:21.455735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.455716    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:21.507456 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.507380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:21.625002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.624971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:21.836119 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.835997    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerID="0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf" exitCode=0
Sep 10 11:27:21.836119 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.836072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerDied","Data":"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"}
Sep 10 11:27:21.836672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.836641    2578 scope.go:117] "RemoveContainer" containerID="0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"
Sep 10 11:27:21.837660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.837636    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:21.837730 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.837710    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:21.837766 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:21.837741    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:22.092701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:22.092610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:22.248437 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:22.248407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:22.712341 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:22.712305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:22.843896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:22.843859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerStarted","Data":"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"}
Sep 10 11:27:23.430539 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:23.430495    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:23.430724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:23.430680    2578 patch_prober.go:28] interesting pod/e2e-autores-8de26b-openshift-default-774f69876-tkckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 11:27:23.430774 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:23.430732    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 11:27:23.496879 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:23.496831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:24.243810 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:24.243771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:24.430442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:24.430406    2578 patch_prober.go:28] interesting pod/e2e-autores-8de26b-openshift-default-774f69876-tkckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 11:27:24.430618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:24.430468    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 11:27:24.436532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:24.436502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:24.539270 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:24.539184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:25.430267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:25.430230    2578 patch_prober.go:28] interesting pod/e2e-autores-8de26b-openshift-default-774f69876-tkckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 11:27:25.430657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:25.430282    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 11:27:25.515787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:25.515752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:26.109789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:26.109745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:26.430480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:26.430393    2578 patch_prober.go:28] interesting pod/e2e-autores-8de26b-openshift-default-774f69876-tkckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 11:27:26.430480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:26.430450    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 11:27:26.633875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:26.633838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:26.857664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:26.857634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:27.431019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:27.430973    2578 patch_prober.go:28] interesting pod/e2e-autores-8de26b-openshift-default-774f69876-tkckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 11:27:27.431482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:27.431057    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 11:27:28.545680 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:28.545639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:28.547012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:28.546978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:28.549514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:28.549488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:27:37.935929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.935889    2578 generic.go:356] "Generic (PLEG): container finished" podID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerID="9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c" exitCode=0
Sep 10 11:27:37.936493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.935969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerDied","Data":"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"}
Sep 10 11:27:37.936567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.936495    2578 scope.go:117] "RemoveContainer" containerID="9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"
Sep 10 11:27:37.937684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.937652    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:37.937748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.937733    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:37.937794 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:37.937771    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:38.942430 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:38.942394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerStarted","Data":"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"}
Sep 10 11:27:39.671490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:39.671439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:39.671785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:39.671761    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 10 11:27:39.671881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:39.671820    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 10 11:27:40.410763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:40.410681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:40.671808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:40.671717    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 10 11:27:40.671808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:40.671775    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 10 11:27:41.293204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.293151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.421448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.421396    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:41.421996 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.421962    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" podUID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" containerName="payload-pre-processing" containerID="cri-o://4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2" gracePeriod=30
Sep 10 11:27:41.475895 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.475861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.581973 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.581891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.671811 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.671768    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 10 11:27:41.672023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.671843    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 10 11:27:41.717289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.717249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.722113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.722076    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:41.811323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.810581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bd8d\" (UniqueName: \"kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d\") pod \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") "
Sep 10 11:27:41.811323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.810740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume\") pod \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\" (UID: \"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd\") "
Sep 10 11:27:41.812065 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.811738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" (UID: "1052de84-b4e4-4a4c-8ac4-65650e0ec9bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:27:41.818219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.817723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d" (OuterVolumeSpecName: "kube-api-access-8bd8d") pod "1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" (UID: "1052de84-b4e4-4a4c-8ac4-65650e0ec9bd"). InnerVolumeSpecName "kube-api-access-8bd8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:27:41.854873 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.854766    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:27:41.855051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.855030    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy" containerID="cri-o://2f45f41fef554b39a11eef0eb82aa60341899fe8f6b90d9a6873486a80313a29" gracePeriod=30
Sep 10 11:27:41.861949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.861924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.911830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.911712    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:41.911830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.911750    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bd8d\" (UniqueName: \"kubernetes.io/projected/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd-kube-api-access-8bd8d\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:41.952925 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.952888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:41.960607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.960300    2578 generic.go:356] "Generic (PLEG): container finished" podID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" containerID="4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2" exitCode=0
Sep 10 11:27:41.960607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.960354    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"
Sep 10 11:27:41.960607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.960405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" event={"ID":"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd","Type":"ContainerDied","Data":"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"}
Sep 10 11:27:41.960607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.960447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8" event={"ID":"1052de84-b4e4-4a4c-8ac4-65650e0ec9bd","Type":"ContainerDied","Data":"3d11fbdb7d30a32e2d1cf88c7a9fd5d649b28e94342619c8eb23ebdf84e2a992"}
Sep 10 11:27:41.960607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.960475    2578 scope.go:117] "RemoveContainer" containerID="4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"
Sep 10 11:27:41.981396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.981352    2578 scope.go:117] "RemoveContainer" containerID="4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"
Sep 10 11:27:41.982109 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:27:41.981874    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2\": container with ID starting with 4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2 not found: ID does not exist" containerID="4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"
Sep 10 11:27:41.982109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:41.981911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2"} err="failed to get container status \"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2\": rpc error: code = NotFound desc = could not find container \"4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2\": container with ID starting with 4e68186ea917198e127193a4d50bd336fd03784655668f787e25392e9ef5a9c2 not found: ID does not exist"
Sep 10 11:27:42.005109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.005039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:42.014389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.014349    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff377da-8dc44cc6b-mjfp8"]
Sep 10 11:27:42.373821 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.373764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:42.457914 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.457882    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" path="/var/lib/kubelet/pods/1052de84-b4e4-4a4c-8ac4-65650e0ec9bd/volumes"
Sep 10 11:27:42.472375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.472278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:42.586012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.585974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:42.672396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.672308    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 10 11:27:42.672396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.672367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 10 11:27:42.691909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.691851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:42.914796 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.914752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:27:42.915635 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.915610    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" containerName="payload-pre-processing"
Sep 10 11:27:42.915794 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.915781    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" containerName="payload-pre-processing"
Sep 10 11:27:42.915985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.915972    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1052de84-b4e4-4a4c-8ac4-65650e0ec9bd" containerName="payload-pre-processing"
Sep 10 11:27:42.930180 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.930101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:42.933339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.933276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-913c8e9c-gw-openshift-default-dockercfg-jfbkq\""
Sep 10 11:27:42.938352 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:42.938326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:27:43.008120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.004392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:43.028466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.028414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.028787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.028765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.028992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029161 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5jf\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029744 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.029902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.029880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5jf\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131621 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.131954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.131894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.132227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.132010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.132227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.132064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.133878 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.133849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.134274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.134254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.144596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.144577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.144675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.144621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5jf\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf\") pod \"e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") " pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.243320 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.243236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:43.316738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.316699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:43.552500 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.552352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:27:43.554892 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:27:43.554865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663498bb_602d_40e2_ae3c_fb0837b9e976.slice/crio-af3988951d0afbf629299a534531e85a1841e3d13377a59a27befe6a7038894e WatchSource:0}: Error finding container af3988951d0afbf629299a534531e85a1841e3d13377a59a27befe6a7038894e: Status 404 returned error can't find the container with id af3988951d0afbf629299a534531e85a1841e3d13377a59a27befe6a7038894e
Sep 10 11:27:43.557290 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.557250    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:43.557377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.557341    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:43.557447 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.557388    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:43.672361 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.672327    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 10 11:27:43.672524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.672389    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 10 11:27:43.972852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.972807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerStarted","Data":"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"}
Sep 10 11:27:43.973038 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.972870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerStarted","Data":"af3988951d0afbf629299a534531e85a1841e3d13377a59a27befe6a7038894e"}
Sep 10 11:27:43.994668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:43.994554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podStartSLOduration=1.9945314600000001 podStartE2EDuration="1.99453146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:27:43.994344441 +0000 UTC m=+782.069670117" watchObservedRunningTime="2026-09-10 11:27:43.99453146 +0000 UTC m=+782.069857164"
Sep 10 11:27:44.244216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.244120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:44.249112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.249054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:44.676013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.675984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:44.676408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.676316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:44.677077 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.677057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:27:44.803207 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.803181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:44.978342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.978256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:44.979253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:44.979234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:45.492578 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:45.492528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:45.601178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:45.601145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:45.789276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:45.789194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:45.903440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:45.903410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:46.520302 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:46.520271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:46.644935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:46.644904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:27:46.991300 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:46.991268    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerID="2f45f41fef554b39a11eef0eb82aa60341899fe8f6b90d9a6873486a80313a29" exitCode=0
Sep 10 11:27:46.991754 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:46.991339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerDied","Data":"2f45f41fef554b39a11eef0eb82aa60341899fe8f6b90d9a6873486a80313a29"}
Sep 10 11:27:46.991754 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:46.991432    2578 scope.go:117] "RemoveContainer" containerID="2690906dd1e6efdc12762ff5911fd71a3cc5c2032e7c6e15ec2072226971dd40"
Sep 10 11:27:47.110360 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.110335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:47.174496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174410    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gxbs\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174653    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.174713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data\") pod \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\" (UID: \"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567\") "
Sep 10 11:27:47.175110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:27:47.175110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.174893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:27:47.175110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.175074    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.175276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.175127    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.175276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.175149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:27:47.175276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.175185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:27:47.175276 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.175250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:27:47.176868 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.176847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:27:47.177052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.177039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:27:47.177327 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.177299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs" (OuterVolumeSpecName: "kube-api-access-5gxbs") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "kube-api-access-5gxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:27:47.177327 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.177315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" (UID: "2c38e4f4-c726-4ac4-bf94-ed8fd83c1567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:27:47.276462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276408    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxbs\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-kube-api-access-5gxbs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276455    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276468    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276481    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276492    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276503    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.276714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.276514    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:27:47.998253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.998223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"
Sep 10 11:27:47.998727 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.998256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz" event={"ID":"2c38e4f4-c726-4ac4-bf94-ed8fd83c1567","Type":"ContainerDied","Data":"f03ae3340df7e814c06165b47e27506b77d93d93443d6bf58a6917676ba89c98"}
Sep 10 11:27:47.998727 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:47.998320    2578 scope.go:117] "RemoveContainer" containerID="2f45f41fef554b39a11eef0eb82aa60341899fe8f6b90d9a6873486a80313a29"
Sep 10 11:27:48.021729 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:48.021704    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:27:48.024863 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:48.024837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aff377da-openshift-default-5dcf85466-fkzsz"]
Sep 10 11:27:48.455989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:48.455957    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" path="/var/lib/kubelet/pods/2c38e4f4-c726-4ac4-bf94-ed8fd83c1567/volumes"
Sep 10 11:27:58.056689 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.056649    2578 generic.go:356] "Generic (PLEG): container finished" podID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerID="da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899" exitCode=0
Sep 10 11:27:58.057240 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.056727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerDied","Data":"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"}
Sep 10 11:27:58.057240 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.057213    2578 scope.go:117] "RemoveContainer" containerID="da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"
Sep 10 11:27:58.058293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.058262    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:58.058388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.058329    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:58.058388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:58.058364    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:27:59.063847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:59.063812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerStarted","Data":"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"}
Sep 10 11:27:59.244509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:59.244465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:27:59.245664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:59.245640    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 11:27:59.245735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:59.245691    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 11:27:59.978903 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:27:59.978867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:28:00.244380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:00.244294    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 11:28:00.244380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:00.244354    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 11:28:01.244499 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:01.244463    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 11:28:01.244913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:01.244520    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 11:28:02.243982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:02.243947    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 11:28:02.244192 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:02.244014    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 11:28:03.243953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:03.243917    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 11:28:03.244423 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:03.243975    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 11:28:04.271241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:04.271205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:28:04.272355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:04.272333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:28:10.524640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.524598    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"]
Sep 10 11:28:10.525175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525155    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.525236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525179    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.525236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525195    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.525236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525204    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.525333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525320    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.525367 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.525333    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c38e4f4-c726-4ac4-bf94-ed8fd83c1567" containerName="istio-proxy"
Sep 10 11:28:10.528917 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.528896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:10.533051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.533031    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:28:10.533253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.533233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:28:10.542503 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.542474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"]
Sep 10 11:28:10.610107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.610064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj\" (UID: \"a3db777e-f097-4913-ad9c-023b7f2fbb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:10.710620 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.710582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj\" (UID: \"a3db777e-f097-4913-ad9c-023b7f2fbb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:10.711239 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.711219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj\" (UID: \"a3db777e-f097-4913-ad9c-023b7f2fbb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:10.843904 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:10.843821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:11.123285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:11.123253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"]
Sep 10 11:28:11.124879 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:28:11.124855    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3db777e_f097_4913_ad9c_023b7f2fbb51.slice/crio-a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6 WatchSource:0}: Error finding container a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6: Status 404 returned error can't find the container with id a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6
Sep 10 11:28:12.135827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:12.135784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj" event={"ID":"a3db777e-f097-4913-ad9c-023b7f2fbb51","Type":"ContainerStarted","Data":"a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6"}
Sep 10 11:28:14.148435 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:14.148394    2578 generic.go:356] "Generic (PLEG): container finished" podID="a3db777e-f097-4913-ad9c-023b7f2fbb51" containerID="37b1436621149281033c020a4d6fcf51cfe0a0b0e658a4590ca85159b678827d" exitCode=0
Sep 10 11:28:14.148816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:14.148530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj" event={"ID":"a3db777e-f097-4913-ad9c-023b7f2fbb51","Type":"ContainerDied","Data":"37b1436621149281033c020a4d6fcf51cfe0a0b0e658a4590ca85159b678827d"}
Sep 10 11:28:15.291799 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:15.291774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:15.351901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:15.351868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca\") pod \"a3db777e-f097-4913-ad9c-023b7f2fbb51\" (UID: \"a3db777e-f097-4913-ad9c-023b7f2fbb51\") "
Sep 10 11:28:15.352256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:15.352236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3db777e-f097-4913-ad9c-023b7f2fbb51" (UID: "a3db777e-f097-4913-ad9c-023b7f2fbb51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:28:15.453230 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:15.453145    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db777e-f097-4913-ad9c-023b7f2fbb51-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:16.165868 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:16.165836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"
Sep 10 11:28:16.166063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:16.165866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj" event={"ID":"a3db777e-f097-4913-ad9c-023b7f2fbb51","Type":"ContainerDied","Data":"a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6"}
Sep 10 11:28:16.166063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:16.165966    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57c8c2636b7c57e818a460fa4559fb384497fe38eefcfa15b23280f8a936bf6"
Sep 10 11:28:17.150762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.150723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:17.336596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.336556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:17.450511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.450412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:17.552803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.552768    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"]
Sep 10 11:28:17.553022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.552973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:17.555371 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.555347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-913c8e9c-fx8gj"]
Sep 10 11:28:17.652531 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.652499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:17.695725 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.695690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:28:17.695920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.695899    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" containerID="cri-o://2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130" gracePeriod=30
Sep 10 11:28:17.741248 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:17.741179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:18.156960 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.156929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:18.442541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.442462    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:28:18.442910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.442892    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3db777e-f097-4913-ad9c-023b7f2fbb51" containerName="revoke-keys"
Sep 10 11:28:18.443005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.442912    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3db777e-f097-4913-ad9c-023b7f2fbb51" containerName="revoke-keys"
Sep 10 11:28:18.443062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.443027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3db777e-f097-4913-ad9c-023b7f2fbb51" containerName="revoke-keys"
Sep 10 11:28:18.446224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.446205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.449344 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.449324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-54215d39-openshift-default-dockercfg-ffbnn\""
Sep 10 11:28:18.456418 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.456396    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3db777e-f097-4913-ad9c-023b7f2fbb51" path="/var/lib/kubelet/pods/a3db777e-f097-4913-ad9c-023b7f2fbb51/volumes"
Sep 10 11:28:18.459811 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.459789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:28:18.481993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.481972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:18.483210 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f52m\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.483502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.483473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f52m\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.584990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.584977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.585704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.585685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.587640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.587620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.588350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.588328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.594038 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.594016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.594256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.594239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f52m\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m\") pod \"e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") " pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:18.718513 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.718398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:18.757063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:18.757037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:19.040570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.040543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:28:19.041488 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:28:19.041465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec4e2de_2aac_42d1_a2da_24f56095c8de.slice/crio-55bceeaf082ef52a8437cf208c6cc8e888cf6316061cb3e65664693360013fd9 WatchSource:0}: Error finding container 55bceeaf082ef52a8437cf208c6cc8e888cf6316061cb3e65664693360013fd9: Status 404 returned error can't find the container with id 55bceeaf082ef52a8437cf208c6cc8e888cf6316061cb3e65664693360013fd9
Sep 10 11:28:19.043820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.043785    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:19.043924 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.043861    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:19.043924 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.043903    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:19.183877 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.183840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerStarted","Data":"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"}
Sep 10 11:28:19.184377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.183906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerStarted","Data":"55bceeaf082ef52a8437cf208c6cc8e888cf6316061cb3e65664693360013fd9"}
Sep 10 11:28:19.207869 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.207811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podStartSLOduration=1.207787613 podStartE2EDuration="1.207787613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:28:19.205771616 +0000 UTC m=+817.281097283" watchObservedRunningTime="2026-09-10 11:28:19.207787613 +0000 UTC m=+817.283113277"
Sep 10 11:28:19.273809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.273771    2578 patch_prober.go:28] interesting pod/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:28:19.273989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.273834    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:28:19.757545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:19.757510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:20.597188 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:20.597152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:20.762611 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:20.762581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:20.773932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:20.773908    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:21.195149 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:21.195118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:21.196011 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:21.195987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:21.880974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:21.880937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:21.948073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:21.948025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:22.393881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:22.393846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:22.651980 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:22.651872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:22.752712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:22.752682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:22.982870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:22.982847    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:28:23.130172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130395 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz5jf\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130545 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130490    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:23.130707 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130707 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs\") pod \"663498bb-602d-40e2-ae3c-fb0837b9e976\" (UID: \"663498bb-602d-40e2-ae3c-fb0837b9e976\") "
Sep 10 11:28:23.130832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:23.130832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data" (OuterVolumeSpecName: "istio-data") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:23.130987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.130896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:28:23.131156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.131129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:23.131329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.131156    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663498bb-602d-40e2-ae3c-fb0837b9e976-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.131612 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.131591    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.131705 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.131616    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.131705 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.131632    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.132781 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.132755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:28:23.132912 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.132878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:23.133226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.133206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf" (OuterVolumeSpecName: "kube-api-access-cz5jf") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "kube-api-access-cz5jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:28:23.133293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.133221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token" (OuterVolumeSpecName: "istio-token") pod "663498bb-602d-40e2-ae3c-fb0837b9e976" (UID: "663498bb-602d-40e2-ae3c-fb0837b9e976"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:28:23.205575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.205540    2578 generic.go:356] "Generic (PLEG): container finished" podID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerID="2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130" exitCode=0
Sep 10 11:28:23.205780 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.205589    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"
Sep 10 11:28:23.205780 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.205637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerDied","Data":"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"}
Sep 10 11:28:23.205780 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.205699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5" event={"ID":"663498bb-602d-40e2-ae3c-fb0837b9e976","Type":"ContainerDied","Data":"af3988951d0afbf629299a534531e85a1841e3d13377a59a27befe6a7038894e"}
Sep 10 11:28:23.205780 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.205728    2578 scope.go:117] "RemoveContainer" containerID="2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"
Sep 10 11:28:23.216736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.216718    2578 scope.go:117] "RemoveContainer" containerID="da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"
Sep 10 11:28:23.225470 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.225452    2578 scope.go:117] "RemoveContainer" containerID="2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"
Sep 10 11:28:23.225744 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:28:23.225724    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130\": container with ID starting with 2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130 not found: ID does not exist" containerID="2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"
Sep 10 11:28:23.225808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.225751    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130"} err="failed to get container status \"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130\": rpc error: code = NotFound desc = could not find container \"2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130\": container with ID starting with 2fc52d82ca32067914d951a9cbc4e113edffebb82b7b4e7828e9acc504cbc130 not found: ID does not exist"
Sep 10 11:28:23.225808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.225768    2578 scope.go:117] "RemoveContainer" containerID="da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"
Sep 10 11:28:23.226000 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:28:23.225986    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899\": container with ID starting with da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899 not found: ID does not exist" containerID="da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"
Sep 10 11:28:23.226099 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.226003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899"} err="failed to get container status \"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899\": rpc error: code = NotFound desc = could not find container \"da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899\": container with ID starting with da6131ebea4bf58aa69140f1595f4f4a39bd003d5b3e8c59738341d13ccc1899 not found: ID does not exist"
Sep 10 11:28:23.229466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.229442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:28:23.232530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.232509    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.232617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.232532    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5jf\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-kube-api-access-cz5jf\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.232617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.232548    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.232617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.232560    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663498bb-602d-40e2-ae3c-fb0837b9e976-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.232617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.232571    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663498bb-602d-40e2-ae3c-fb0837b9e976-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:23.233448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:23.233430    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-913c8e9c-gw-openshift-default-75d474d6fb-7xrj5"]
Sep 10 11:28:24.456871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:24.456828    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" path="/var/lib/kubelet/pods/663498bb-602d-40e2-ae3c-fb0837b9e976/volumes"
Sep 10 11:28:36.196160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.196121    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:36.196555 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.196192    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:36.285496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.285462    2578 generic.go:356] "Generic (PLEG): container finished" podID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerID="df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a" exitCode=0
Sep 10 11:28:36.285702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.285535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerDied","Data":"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"}
Sep 10 11:28:36.286052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.286034    2578 scope.go:117] "RemoveContainer" containerID="df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"
Sep 10 11:28:36.287504 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.287467    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:36.287583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.287565    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:36.287631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.287616    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:36.464037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:36.463962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:37.302047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:37.302000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerStarted","Data":"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"}
Sep 10 11:28:37.757977 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:37.757932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:37.758182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:37.758155    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:37.758235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:37.758202    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:38.758184 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:38.758151    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:38.758582 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:38.758206    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:39.757455 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:39.757419    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:39.757629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:39.757491    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:40.757905 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:40.757870    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:40.758313 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:40.757923    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:41.758366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:41.758323    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 11:28:41.758844 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:41.758413    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 11:28:42.762118 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:42.762011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:42.762489 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:42.762294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:42.763122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:42.763103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:46.388204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.388156    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"]
Sep 10 11:28:46.388687 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.388673    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:46.388738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.388689    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:46.388773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.388759    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:46.388773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.388766    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:46.395894 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.395867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:46.401295 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.401267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:28:46.401442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.401271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:28:46.405669 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.405641    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"]
Sep 10 11:28:46.572933 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.572893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54215d39-kqnkm\" (UID: \"2db9a7a7-a7b4-47e9-a131-700664e5dd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:46.673562 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.673472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54215d39-kqnkm\" (UID: \"2db9a7a7-a7b4-47e9-a131-700664e5dd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:46.674144 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.674124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54215d39-kqnkm\" (UID: \"2db9a7a7-a7b4-47e9-a131-700664e5dd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:46.707459 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.707425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:46.982081 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:46.982057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"]
Sep 10 11:28:46.983624 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:28:46.983598    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db9a7a7_a7b4_47e9_a131_700664e5dd3a.slice/crio-e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650 WatchSource:0}: Error finding container e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650: Status 404 returned error can't find the container with id e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650
Sep 10 11:28:47.365141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:47.365105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm" event={"ID":"2db9a7a7-a7b4-47e9-a131-700664e5dd3a","Type":"ContainerStarted","Data":"1f93365ef6da13476d08619dca8324e31f5dffcb90ae8e5525f0cf09c8cc2ac4"}
Sep 10 11:28:47.365338 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:47.365170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm" event={"ID":"2db9a7a7-a7b4-47e9-a131-700664e5dd3a","Type":"ContainerStarted","Data":"e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650"}
Sep 10 11:28:47.381992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:47.381945    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm" podStartSLOduration=1.381929095 podStartE2EDuration="1.381929095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:28:47.381128551 +0000 UTC m=+845.456454213" watchObservedRunningTime="2026-09-10 11:28:47.381929095 +0000 UTC m=+845.457254782"
Sep 10 11:28:48.372976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:48.372938    2578 generic.go:356] "Generic (PLEG): container finished" podID="2db9a7a7-a7b4-47e9-a131-700664e5dd3a" containerID="1f93365ef6da13476d08619dca8324e31f5dffcb90ae8e5525f0cf09c8cc2ac4" exitCode=0
Sep 10 11:28:48.372976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:48.372976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm" event={"ID":"2db9a7a7-a7b4-47e9-a131-700664e5dd3a","Type":"ContainerDied","Data":"1f93365ef6da13476d08619dca8324e31f5dffcb90ae8e5525f0cf09c8cc2ac4"}
Sep 10 11:28:49.511396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.511371    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:49.601927 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.601890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca\") pod \"2db9a7a7-a7b4-47e9-a131-700664e5dd3a\" (UID: \"2db9a7a7-a7b4-47e9-a131-700664e5dd3a\") "
Sep 10 11:28:49.602214 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.602191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2db9a7a7-a7b4-47e9-a131-700664e5dd3a" (UID: "2db9a7a7-a7b4-47e9-a131-700664e5dd3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:28:49.702971 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.702865    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db9a7a7-a7b4-47e9-a131-700664e5dd3a-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:49.852721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.852677    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:28:49.853139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.853122    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:49.853241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.853141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="663498bb-602d-40e2-ae3c-fb0837b9e976" containerName="istio-proxy"
Sep 10 11:28:49.853241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.853162    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db9a7a7-a7b4-47e9-a131-700664e5dd3a" containerName="revoke-keys"
Sep 10 11:28:49.853241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.853170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db9a7a7-a7b4-47e9-a131-700664e5dd3a" containerName="revoke-keys"
Sep 10 11:28:49.853389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.853275    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db9a7a7-a7b4-47e9-a131-700664e5dd3a" containerName="revoke-keys"
Sep 10 11:28:49.856512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.856492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.861193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.861163    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8de26b\"/\"default-dockercfg-zpc6n\""
Sep 10 11:28:49.861193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.861183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8de26b\"/\"autores-model-8de26b-kserve-self-signed-certs\""
Sep 10 11:28:49.861358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.861193    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8de26b\"/\"openshift-service-ca.crt\""
Sep 10 11:28:49.861358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.861164    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8de26b\"/\"kube-root-ca.crt\""
Sep 10 11:28:49.869238 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.869213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:28:49.904110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.904321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.904321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhfl\" (UniqueName: \"kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.904321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.904429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:49.904429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:49.904393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhfl\" (UniqueName: \"kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.005893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.005747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.006148 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.006068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.006148 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.006130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.006323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.006300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.008158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.008131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.008681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.008659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.016814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.016765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhfl\" (UniqueName: \"kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl\") pod \"autores-model-8de26b-kserve-5d8766d5cf-jkdt9\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") " pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.047304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.047216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:50.166393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.166355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:28:50.386226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.386197    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"
Sep 10 11:28:50.386226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.386202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm" event={"ID":"2db9a7a7-a7b4-47e9-a131-700664e5dd3a","Type":"ContainerDied","Data":"e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650"}
Sep 10 11:28:50.386490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.386278    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8425064cdea4094573510d619a60747bd4f3452179644cf141f93ffca9bc650"
Sep 10 11:28:50.446227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:50.446193    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:28:50.447757 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:28:50.447727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84650abd_0659_40f8_bf84_51a9bb89b8c7.slice/crio-ee49a203cae51a2f60ab01c0863cee4010daa73031a1552a63bf6c3079df1288 WatchSource:0}: Error finding container ee49a203cae51a2f60ab01c0863cee4010daa73031a1552a63bf6c3079df1288: Status 404 returned error can't find the container with id ee49a203cae51a2f60ab01c0863cee4010daa73031a1552a63bf6c3079df1288
Sep 10 11:28:51.394826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:51.394782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" event={"ID":"84650abd-0659-40f8-bf84-51a9bb89b8c7","Type":"ContainerStarted","Data":"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"}
Sep 10 11:28:51.395229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:51.394836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" event={"ID":"84650abd-0659-40f8-bf84-51a9bb89b8c7","Type":"ContainerStarted","Data":"ee49a203cae51a2f60ab01c0863cee4010daa73031a1552a63bf6c3079df1288"}
Sep 10 11:28:51.415181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:51.415132    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" podStartSLOduration=2.4151173 podStartE2EDuration="2.4151173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:28:51.414415609 +0000 UTC m=+849.489741273" watchObservedRunningTime="2026-09-10 11:28:51.4151173 +0000 UTC m=+849.490442961"
Sep 10 11:28:52.394624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.394588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:52.479967 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.479931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:52.577998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.577967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:52.640870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.640837    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"]
Sep 10 11:28:52.646943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.646879    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54215d39-kqnkm"]
Sep 10 11:28:52.667364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:52.667332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:53.836068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:53.836035    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:28:53.836556 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:53.836381    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" containerID="cri-o://c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926" gracePeriod=30
Sep 10 11:28:53.888830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:53.888791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:54.455295 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.455263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db9a7a7-a7b4-47e9-a131-700664e5dd3a" path="/var/lib/kubelet/pods/2db9a7a7-a7b4-47e9-a131-700664e5dd3a/volumes"
Sep 10 11:28:54.478425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.478393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:54.645310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.645276    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:28:54.655430 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.655396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.658594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.658568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0024ffd2-openshift-default-dockercfg-9znr6\""
Sep 10 11:28:54.666785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.666760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:28:54.668670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.668643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.750788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.750862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.750940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrjs\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.750987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.751054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.751113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.751152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.751189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.751657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.751220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.851826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.851790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrjs\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.851826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.851834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.851892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.851924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.852991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.852965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.853165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.853046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.853264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.853176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.853264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.853193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.853420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.853311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.855081 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.855052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.855400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.855372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.862249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.862227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrjs\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.862617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.862596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token\") pod \"e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") " pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:54.969402 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:54.969369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:55.004270 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.004166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:55.256260 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.256223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:28:55.259023 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:28:55.258999    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca54ad06_d2f9_4663_bcdb_c74167ffee10.slice/crio-25c9f0dbde469c208b862d5af4ea5137d0d779733073caad3af1773cd6e07853 WatchSource:0}: Error finding container 25c9f0dbde469c208b862d5af4ea5137d0d779733073caad3af1773cd6e07853: Status 404 returned error can't find the container with id 25c9f0dbde469c208b862d5af4ea5137d0d779733073caad3af1773cd6e07853
Sep 10 11:28:55.261372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.261341    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:55.261429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.261411    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:55.261462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.261440    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:28:55.417438 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.417397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerStarted","Data":"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"}
Sep 10 11:28:55.417595 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.417466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerStarted","Data":"25c9f0dbde469c208b862d5af4ea5137d0d779733073caad3af1773cd6e07853"}
Sep 10 11:28:55.441580 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.441531    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podStartSLOduration=1.441513767 podStartE2EDuration="1.441513767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:28:55.440417696 +0000 UTC m=+853.515743358" watchObservedRunningTime="2026-09-10 11:28:55.441513767 +0000 UTC m=+853.516839430"
Sep 10 11:28:55.969593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:55.969556    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:56.832492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:56.832456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:56.922067 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:56.922035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:56.975726 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:56.975695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:57.429326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:57.429295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:57.435593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:57.435568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:28:57.765166 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:57.765047    2578 patch_prober.go:28] interesting pod/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:28:57.765320 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:57.765157    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:28:58.094150 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:58.094026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:58.191537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:58.191498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:58.418779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:58.418730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:58.719162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:58.719017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:58.814866 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:58.814838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:28:59.118579 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.118556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:59.194316 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194316 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f52m\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket\") pod \"aec4e2de-2aac-42d1-a2da-24f56095c8de\" (UID: \"aec4e2de-2aac-42d1-a2da-24f56095c8de\") "
Sep 10 11:28:59.194831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data" (OuterVolumeSpecName: "istio-data") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:59.194831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:59.195042 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:28:59.195042 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.194976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:59.195172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.195112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:59.195172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.195118    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.195172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.195138    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.195172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.195152    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.195172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.195164    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec4e2de-2aac-42d1-a2da-24f56095c8de-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.197398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.197347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:28:59.197527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.197421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:28:59.197527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.197435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token" (OuterVolumeSpecName: "istio-token") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:28:59.197527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.197458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m" (OuterVolumeSpecName: "kube-api-access-5f52m") pod "aec4e2de-2aac-42d1-a2da-24f56095c8de" (UID: "aec4e2de-2aac-42d1-a2da-24f56095c8de"). InnerVolumeSpecName "kube-api-access-5f52m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:28:59.296570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.296479    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.296570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.296510    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.296570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.296522    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f52m\" (UniqueName: \"kubernetes.io/projected/aec4e2de-2aac-42d1-a2da-24f56095c8de-kube-api-access-5f52m\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.296570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.296533    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec4e2de-2aac-42d1-a2da-24f56095c8de-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.296570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.296545    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec4e2de-2aac-42d1-a2da-24f56095c8de-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:28:59.440062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.440025    2578 generic.go:356] "Generic (PLEG): container finished" podID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerID="c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926" exitCode=0
Sep 10 11:28:59.440271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.440111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"
Sep 10 11:28:59.440271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.440116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerDied","Data":"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"}
Sep 10 11:28:59.440271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.440174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk" event={"ID":"aec4e2de-2aac-42d1-a2da-24f56095c8de","Type":"ContainerDied","Data":"55bceeaf082ef52a8437cf208c6cc8e888cf6316061cb3e65664693360013fd9"}
Sep 10 11:28:59.440271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.440190    2578 scope.go:117] "RemoveContainer" containerID="c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"
Sep 10 11:28:59.449884 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.449867    2578 scope.go:117] "RemoveContainer" containerID="df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"
Sep 10 11:28:59.457899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.457885    2578 scope.go:117] "RemoveContainer" containerID="c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"
Sep 10 11:28:59.458173 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:28:59.458154    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926\": container with ID starting with c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926 not found: ID does not exist" containerID="c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"
Sep 10 11:28:59.458219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.458182    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926"} err="failed to get container status \"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926\": rpc error: code = NotFound desc = could not find container \"c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926\": container with ID starting with c40ea643981505290baa54a4b8c3988bc6de8c9cc2a644a4cfff51106c91e926 not found: ID does not exist"
Sep 10 11:28:59.458219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.458201    2578 scope.go:117] "RemoveContainer" containerID="df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"
Sep 10 11:28:59.458449 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:28:59.458433    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a\": container with ID starting with df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a not found: ID does not exist" containerID="df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"
Sep 10 11:28:59.458488 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.458453    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a"} err="failed to get container status \"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a\": rpc error: code = NotFound desc = could not find container \"df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a\": container with ID starting with df8a9be46e304a6b2a8a792adb5d9ea81552e8dc07cbbe66577cde273761e63a not found: ID does not exist"
Sep 10 11:28:59.485901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.485876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:28:59.497542 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:28:59.497513    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-54215d39-openshift-default-684d64c59d-j8bbk"]
Sep 10 11:29:00.167541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:00.167506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:00.168416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:00.168392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:00.183072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:00.183046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:00.456601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:00.456510    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" path="/var/lib/kubelet/pods/aec4e2de-2aac-42d1-a2da-24f56095c8de/volumes"
Sep 10 11:29:00.459360 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:00.459337    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:05.989468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:05.989423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:06.091950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:06.091906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:12.430287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.430236    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:12.430638 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.430297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:12.521165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.521129    2578 generic.go:356] "Generic (PLEG): container finished" podID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerID="bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d" exitCode=0
Sep 10 11:29:12.521331 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.521204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerDied","Data":"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"}
Sep 10 11:29:12.521611 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.521598    2578 scope.go:117] "RemoveContainer" containerID="bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"
Sep 10 11:29:12.522687 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.522657    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:12.522787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.522742    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:12.522859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:12.522819    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:13.095658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.095619    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:29:13.096199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.096199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.096391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.096391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096219    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.096495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096403    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.096495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.096417    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec4e2de-2aac-42d1-a2da-24f56095c8de" containerName="istio-proxy"
Sep 10 11:29:13.099563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.099544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.104730 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.104709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8355df\"/\"openshift-service-ca.crt\""
Sep 10 11:29:13.104843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.104741    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8355df\"/\"default-dockercfg-tkvpv\""
Sep 10 11:29:13.104843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.104708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8355df\"/\"test-model-8355df-kserve-self-signed-certs\""
Sep 10 11:29:13.104843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.104765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8355df\"/\"kube-root-ca.crt\""
Sep 10 11:29:13.112599 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.112571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:29:13.212574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.212535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:13.237848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.237815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.237848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.237853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.238129 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.237941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2nj\" (UniqueName: \"kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.238129 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.238016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.238129 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.238049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.238252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.238143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2nj\" (UniqueName: \"kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.339864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.339853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.340235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.340215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.340303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.340242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.340303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.340293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.342233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.342200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.342632 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.342615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.356171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.356110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2nj\" (UniqueName: \"kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj\") pod \"test-model-8355df-kserve-8688d4d6dc-6m447\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") " pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.412711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.412324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:13.537425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.537357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerStarted","Data":"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"}
Sep 10 11:29:13.746247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.746222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:29:13.748443 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:29:13.748411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd926c82_dcce_48e2_9454_6a444fbb5ebe.slice/crio-c27b4a37b04f9fc1b28ff8cfa2918bfdbbcf209e2549af331473a910d777cb83 WatchSource:0}: Error finding container c27b4a37b04f9fc1b28ff8cfa2918bfdbbcf209e2549af331473a910d777cb83: Status 404 returned error can't find the container with id c27b4a37b04f9fc1b28ff8cfa2918bfdbbcf209e2549af331473a910d777cb83
Sep 10 11:29:13.970597 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.970555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:13.970838 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.970816    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:13.970890 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:13.970870    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:14.544984 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:14.544941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" event={"ID":"fd926c82-dcce-48e2-9454-6a444fbb5ebe","Type":"ContainerStarted","Data":"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"}
Sep 10 11:29:14.544984 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:14.544990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" event={"ID":"fd926c82-dcce-48e2-9454-6a444fbb5ebe","Type":"ContainerStarted","Data":"c27b4a37b04f9fc1b28ff8cfa2918bfdbbcf209e2549af331473a910d777cb83"}
Sep 10 11:29:14.572107 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:14.572041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" podStartSLOduration=1.572025896 podStartE2EDuration="1.572025896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:29:14.571679814 +0000 UTC m=+872.647005491" watchObservedRunningTime="2026-09-10 11:29:14.572025896 +0000 UTC m=+872.647351560"
Sep 10 11:29:14.969775 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:14.969739    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:14.969955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:14.969808    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:15.700131 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:15.700069    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:29:15.732377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:15.732339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:15.970672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:15.970582    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:15.970672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:15.970640    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:16.577478 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:16.577442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:16.970746 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:16.970712    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:16.971194 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:16.970764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:17.213514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.213468    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:29:17.295597 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.295505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:17.375624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.375590    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"]
Sep 10 11:29:17.378656 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.378640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:17.382390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.382367    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:29:17.382842 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.382817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:29:17.403232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.403200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"]
Sep 10 11:29:17.483908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.483871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8de26b-zhnkf\" (UID: \"90122afc-9d3f-4436-8448-972985506b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:17.584681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.584586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8de26b-zhnkf\" (UID: \"90122afc-9d3f-4436-8448-972985506b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:17.585864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.585632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8de26b-zhnkf\" (UID: \"90122afc-9d3f-4436-8448-972985506b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:17.691428 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.691396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:17.707968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.707940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:17.970170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.970132    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:17.970330 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.970186    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:17.998178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:17.996863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"]
Sep 10 11:29:18.000187 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:29:17.999218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90122afc_9d3f_4436_8448_972985506b74.slice/crio-9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff WatchSource:0}: Error finding container 9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff: Status 404 returned error can't find the container with id 9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff
Sep 10 11:29:18.569043 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:18.568953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf" event={"ID":"90122afc-9d3f-4436-8448-972985506b74","Type":"ContainerStarted","Data":"d1a4ed43964e31b1f7dcec83104134ed4b688854208ab65c35584f4114b5b224"}
Sep 10 11:29:18.569043 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:18.569003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf" event={"ID":"90122afc-9d3f-4436-8448-972985506b74","Type":"ContainerStarted","Data":"9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff"}
Sep 10 11:29:18.606916 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:18.606835    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf" podStartSLOduration=1.606809842 podStartE2EDuration="1.606809842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:29:18.603598056 +0000 UTC m=+876.678923721" watchObservedRunningTime="2026-09-10 11:29:18.606809842 +0000 UTC m=+876.682135525"
Sep 10 11:29:19.037982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.037940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:19.038488 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.038402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:19.039764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.039744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:19.426746 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.426718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:19.576648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.576618    2578 generic.go:356] "Generic (PLEG): container finished" podID="90122afc-9d3f-4436-8448-972985506b74" containerID="d1a4ed43964e31b1f7dcec83104134ed4b688854208ab65c35584f4114b5b224" exitCode=0
Sep 10 11:29:19.576848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:19.576692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf" event={"ID":"90122afc-9d3f-4436-8448-972985506b74","Type":"ContainerDied","Data":"d1a4ed43964e31b1f7dcec83104134ed4b688854208ab65c35584f4114b5b224"}
Sep 10 11:29:20.721134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:20.721110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:20.841308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:20.841218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca\") pod \"90122afc-9d3f-4436-8448-972985506b74\" (UID: \"90122afc-9d3f-4436-8448-972985506b74\") "
Sep 10 11:29:20.841614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:20.841586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90122afc-9d3f-4436-8448-972985506b74" (UID: "90122afc-9d3f-4436-8448-972985506b74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:29:20.841731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:20.841714    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90122afc-9d3f-4436-8448-972985506b74-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:21.589697 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.589663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf" event={"ID":"90122afc-9d3f-4436-8448-972985506b74","Type":"ContainerDied","Data":"9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff"}
Sep 10 11:29:21.589697 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.589696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"
Sep 10 11:29:21.589949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.589710    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cd072e6d91c6b1b61c7224f69cda495e3bdd21626fd2e1c84b8c8b33c073fff"
Sep 10 11:29:21.731519 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.731479    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"]
Sep 10 11:29:21.731901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.731879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90122afc-9d3f-4436-8448-972985506b74" containerName="revoke-keys"
Sep 10 11:29:21.731901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.731890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90122afc-9d3f-4436-8448-972985506b74" containerName="revoke-keys"
Sep 10 11:29:21.731975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.731946    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90122afc-9d3f-4436-8448-972985506b74" containerName="revoke-keys"
Sep 10 11:29:21.735008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.734993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:21.739134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.739107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:29:21.739765 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.739748    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:29:21.746378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.746354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"]
Sep 10 11:29:21.852869 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.852772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c\" (UID: \"4172cd5e-d237-470b-848c-12c4ef327ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:21.955468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.954448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c\" (UID: \"4172cd5e-d237-470b-848c-12c4ef327ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:21.955468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:21.955412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c\" (UID: \"4172cd5e-d237-470b-848c-12c4ef327ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:22.047214 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:22.047170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:22.366017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:22.365993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"]
Sep 10 11:29:22.367721 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:29:22.367691    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4172cd5e_d237_470b_848c_12c4ef327ffe.slice/crio-3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216 WatchSource:0}: Error finding container 3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216: Status 404 returned error can't find the container with id 3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216
Sep 10 11:29:22.596870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:22.596837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c" event={"ID":"4172cd5e-d237-470b-848c-12c4ef327ffe","Type":"ContainerStarted","Data":"3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216"}
Sep 10 11:29:23.412990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.412958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:23.413450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.413006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:23.425636 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.425609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:23.603964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.603880    2578 generic.go:356] "Generic (PLEG): container finished" podID="4172cd5e-d237-470b-848c-12c4ef327ffe" containerID="679881f643d231bc08390068ac044deb0b1d9dd9b7c7827dbeb7ff39155af8f4" exitCode=0
Sep 10 11:29:23.603964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.603938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c" event={"ID":"4172cd5e-d237-470b-848c-12c4ef327ffe","Type":"ContainerDied","Data":"679881f643d231bc08390068ac044deb0b1d9dd9b7c7827dbeb7ff39155af8f4"}
Sep 10 11:29:23.615839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.615814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:29:23.861371 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:23.861337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:24.750507 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:24.750479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:24.882393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:24.882301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca\") pod \"4172cd5e-d237-470b-848c-12c4ef327ffe\" (UID: \"4172cd5e-d237-470b-848c-12c4ef327ffe\") "
Sep 10 11:29:24.882662 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:24.882639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4172cd5e-d237-470b-848c-12c4ef327ffe" (UID: "4172cd5e-d237-470b-848c-12c4ef327ffe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:29:24.983773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:24.983742    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4172cd5e-d237-470b-848c-12c4ef327ffe-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:25.616163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:25.616135    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"
Sep 10 11:29:25.616344 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:25.616162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c" event={"ID":"4172cd5e-d237-470b-848c-12c4ef327ffe","Type":"ContainerDied","Data":"3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216"}
Sep 10 11:29:25.616344 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:25.616218    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4c35891edd13c67e8cc0a52e4bb646697f35648e726245472050b9abf67216"
Sep 10 11:29:27.644559 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:27.644527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.076619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.076543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.176760 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.176729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.422536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.422507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.513169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.513137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.634310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.634254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.735630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.735545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.738658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.738633    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"]
Sep 10 11:29:28.740610 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.740587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0024ffd2-77p8c"]
Sep 10 11:29:28.821245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.821214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.892297 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.892262    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:29:28.892522 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.892498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" containerID="cri-o://1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656" gracePeriod=30
Sep 10 11:29:28.929783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.929748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:28.976936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.976894    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"]
Sep 10 11:29:28.983037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:28.983008    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8de26b-zhnkf"]
Sep 10 11:29:29.024893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:29.024795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:29.351365 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:29.351275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:29.996702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:29.996672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:30.095538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:30.095507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:30.456610 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:30.456568    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4172cd5e-d237-470b-848c-12c4ef327ffe" path="/var/lib/kubelet/pods/4172cd5e-d237-470b-848c-12c4ef327ffe/volumes"
Sep 10 11:29:30.456995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:30.456975    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90122afc-9d3f-4436-8448-972985506b74" path="/var/lib/kubelet/pods/90122afc-9d3f-4436-8448-972985506b74/volumes"
Sep 10 11:29:30.726763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:30.726671    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" podUID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" containerName="main" containerID="cri-o://00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e" gracePeriod=45
Sep 10 11:29:30.977684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:30.977614    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:29:31.044450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-895g6\" (UniqueName: \"kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6\") pod \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\" (UID: \"442af6f2-d7bf-4e38-9fbb-7ae6830965ab\") "
Sep 10 11:29:31.044899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.044787    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:31.045280 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.045025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home" (OuterVolumeSpecName: "home") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:31.045280 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.045100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:31.045280 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.045150    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.045280 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.045172    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.047329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.047298    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm" (OuterVolumeSpecName: "dshm") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:31.047479 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.047433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:29:31.047548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.047529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6" (OuterVolumeSpecName: "kube-api-access-895g6") pod "442af6f2-d7bf-4e38-9fbb-7ae6830965ab" (UID: "442af6f2-d7bf-4e38-9fbb-7ae6830965ab"). InnerVolumeSpecName "kube-api-access-895g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:31.146387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.146341    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.146387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.146382    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895g6\" (UniqueName: \"kubernetes.io/projected/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-kube-api-access-895g6\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.146647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.146398    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.146647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.146414    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442af6f2-d7bf-4e38-9fbb-7ae6830965ab-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:31.381668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.381641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:31.657070 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.656986    2578 generic.go:356] "Generic (PLEG): container finished" podID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" containerID="00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e" exitCode=0
Sep 10 11:29:31.657070 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.657035    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"
Sep 10 11:29:31.657275 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.657079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" event={"ID":"442af6f2-d7bf-4e38-9fbb-7ae6830965ab","Type":"ContainerDied","Data":"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"}
Sep 10 11:29:31.657275 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.657134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q" event={"ID":"442af6f2-d7bf-4e38-9fbb-7ae6830965ab","Type":"ContainerDied","Data":"afa8457566e45567c564e29a26ae59558c0613acaf3a26f4027a4220d0d231fc"}
Sep 10 11:29:31.657275 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.657152    2578 scope.go:117] "RemoveContainer" containerID="00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"
Sep 10 11:29:31.666418 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.666399    2578 scope.go:117] "RemoveContainer" containerID="00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"
Sep 10 11:29:31.666721 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:31.666697    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e\": container with ID starting with 00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e not found: ID does not exist" containerID="00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"
Sep 10 11:29:31.666807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.666733    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e"} err="failed to get container status \"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e\": rpc error: code = NotFound desc = could not find container \"00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e\": container with ID starting with 00c109d0e9086635321e1132078887b1c348249560aa4087cd35460f92a90f2e not found: ID does not exist"
Sep 10 11:29:31.698768 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.698744    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:29:31.719079 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:31.719052    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-27987e85-kserve-68ff95c7cd-m5c6q"]
Sep 10 11:29:32.233952 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.233911    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" podUID="84650abd-0659-40f8-bf84-51a9bb89b8c7" containerName="main" containerID="cri-o://a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2" gracePeriod=45
Sep 10 11:29:32.457720 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.457683    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" path="/var/lib/kubelet/pods/442af6f2-d7bf-4e38-9fbb-7ae6830965ab/volumes"
Sep 10 11:29:32.485651 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.485581    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:32.558903 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.558864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.558903 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.558907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.559151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.558958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.559151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.558978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvhfl\" (UniqueName: \"kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.559151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.558997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.559151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs\") pod \"84650abd-0659-40f8-bf84-51a9bb89b8c7\" (UID: \"84650abd-0659-40f8-bf84-51a9bb89b8c7\") "
Sep 10 11:29:32.559377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:32.559377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home" (OuterVolumeSpecName: "home") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:32.559377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:32.559515 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559496    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.559563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559522    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.559563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.559536    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.561428 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.561402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm" (OuterVolumeSpecName: "dshm") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:32.561532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.561512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:29:32.561574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.561544    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl" (OuterVolumeSpecName: "kube-api-access-bvhfl") pod "84650abd-0659-40f8-bf84-51a9bb89b8c7" (UID: "84650abd-0659-40f8-bf84-51a9bb89b8c7"). InnerVolumeSpecName "kube-api-access-bvhfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:32.660377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.660346    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84650abd-0659-40f8-bf84-51a9bb89b8c7-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.660377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.660372    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhfl\" (UniqueName: \"kubernetes.io/projected/84650abd-0659-40f8-bf84-51a9bb89b8c7-kube-api-access-bvhfl\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.660377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.660383    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84650abd-0659-40f8-bf84-51a9bb89b8c7-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:32.663253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.663231    2578 generic.go:356] "Generic (PLEG): container finished" podID="84650abd-0659-40f8-bf84-51a9bb89b8c7" containerID="a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2" exitCode=0
Sep 10 11:29:32.663378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.663275    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"
Sep 10 11:29:32.663378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.663318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" event={"ID":"84650abd-0659-40f8-bf84-51a9bb89b8c7","Type":"ContainerDied","Data":"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"}
Sep 10 11:29:32.663378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.663362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9" event={"ID":"84650abd-0659-40f8-bf84-51a9bb89b8c7","Type":"ContainerDied","Data":"ee49a203cae51a2f60ab01c0863cee4010daa73031a1552a63bf6c3079df1288"}
Sep 10 11:29:32.663582 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.663386    2578 scope.go:117] "RemoveContainer" containerID="a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"
Sep 10 11:29:32.676915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.676895    2578 scope.go:117] "RemoveContainer" containerID="a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"
Sep 10 11:29:32.677243 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:32.677222    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2\": container with ID starting with a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2 not found: ID does not exist" containerID="a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"
Sep 10 11:29:32.677299 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.677251    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2"} err="failed to get container status \"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2\": rpc error: code = NotFound desc = could not find container \"a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2\": container with ID starting with a702620c1675ecd11e7c8a43c46f588ac6199715a27c5fb8dddb272910eb55e2 not found: ID does not exist"
Sep 10 11:29:32.699326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.699301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:29:32.705054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:32.705032    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8de26b/autores-model-8de26b-kserve-5d8766d5cf-jkdt9"]
Sep 10 11:29:34.040287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.040252    2578 patch_prober.go:28] interesting pod/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 11:29:34.040638 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.040325    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 11:29:34.139507 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.139484    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:34.173281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chrjs\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket\") pod \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\" (UID: \"ca54ad06-d2f9-4663-bcdb-c74167ffee10\") "
Sep 10 11:29:34.173830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:34.173830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:29:34.173938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173854    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.173938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173871    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.173938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:34.173938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:34.173938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.173900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:34.175737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.175696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:34.175871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.175788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:29:34.175929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.175883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs" (OuterVolumeSpecName: "kube-api-access-chrjs") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "kube-api-access-chrjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:34.176126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.176079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca54ad06-d2f9-4663-bcdb-c74167ffee10" (UID: "ca54ad06-d2f9-4663-bcdb-c74167ffee10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:34.274770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274737    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.274770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274765    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.274770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274778    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrjs\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-kube-api-access-chrjs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.275007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274788    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.275007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274809    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.275007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274818    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.275007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.274826    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca54ad06-d2f9-4663-bcdb-c74167ffee10-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:34.456466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.456385    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84650abd-0659-40f8-bf84-51a9bb89b8c7" path="/var/lib/kubelet/pods/84650abd-0659-40f8-bf84-51a9bb89b8c7/volumes"
Sep 10 11:29:34.678642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.678608    2578 generic.go:356] "Generic (PLEG): container finished" podID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerID="1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656" exitCode=0
Sep 10 11:29:34.678818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.678664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"
Sep 10 11:29:34.678818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.678685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerDied","Data":"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"}
Sep 10 11:29:34.678818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.678739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k" event={"ID":"ca54ad06-d2f9-4663-bcdb-c74167ffee10","Type":"ContainerDied","Data":"25c9f0dbde469c208b862d5af4ea5137d0d779733073caad3af1773cd6e07853"}
Sep 10 11:29:34.678818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.678761    2578 scope.go:117] "RemoveContainer" containerID="1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"
Sep 10 11:29:34.689002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.688986    2578 scope.go:117] "RemoveContainer" containerID="bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"
Sep 10 11:29:34.697388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.697369    2578 scope.go:117] "RemoveContainer" containerID="1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"
Sep 10 11:29:34.697647 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:34.697626    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656\": container with ID starting with 1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656 not found: ID does not exist" containerID="1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"
Sep 10 11:29:34.697696 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.697653    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656"} err="failed to get container status \"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656\": rpc error: code = NotFound desc = could not find container \"1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656\": container with ID starting with 1d6a95c9f6eece40428313fbb823a66a2198d2754ffc80ee62e4a7a86d61d656 not found: ID does not exist"
Sep 10 11:29:34.697696 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.697672    2578 scope.go:117] "RemoveContainer" containerID="bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"
Sep 10 11:29:34.697901 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:34.697886    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d\": container with ID starting with bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d not found: ID does not exist" containerID="bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"
Sep 10 11:29:34.697938 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.697906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d"} err="failed to get container status \"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d\": rpc error: code = NotFound desc = could not find container \"bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d\": container with ID starting with bb8e4b5b2a97bc89f0b0b3e8525cd0666b0927dee20ceccd8e177301cbff258d not found: ID does not exist"
Sep 10 11:29:34.705856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.705829    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:29:34.707992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:34.707938    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0024ffd2-openshift-default-7d4b6b55fb-b9f5k"]
Sep 10 11:29:35.957794 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:35.957768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:36.455365 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:36.455335    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" path="/var/lib/kubelet/pods/ca54ad06-d2f9-4663-bcdb-c74167ffee10/volumes"
Sep 10 11:29:36.734734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:36.734649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:37.015810 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.015731    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:29:37.016206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.015998    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy" containerID="cri-o://2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc" gracePeriod=30
Sep 10 11:29:37.082717 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.082681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:37.459170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.459132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:37.563411 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.563376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:37.805668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.805586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:29:37.806019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806002    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" containerName="main"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806022    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" containerName="main"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806039    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4172cd5e-d237-470b-848c-12c4ef327ffe" containerName="revoke-keys"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806045    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4172cd5e-d237-470b-848c-12c4ef327ffe" containerName="revoke-keys"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806052    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806060    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:37.806100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806078    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84650abd-0659-40f8-bf84-51a9bb89b8c7" containerName="main"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806103    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84650abd-0659-40f8-bf84-51a9bb89b8c7" containerName="main"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806159    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806167    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4172cd5e-d237-470b-848c-12c4ef327ffe" containerName="revoke-keys"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806178    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="442af6f2-d7bf-4e38-9fbb-7ae6830965ab" containerName="main"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806187    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="84650abd-0659-40f8-bf84-51a9bb89b8c7" containerName="main"
Sep 10 11:29:37.806283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.806192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:37.809219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.809204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.812586 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.812567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f39de1-openshift-default-dockercfg-p66nt\""
Sep 10 11:29:37.827636 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.827609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:29:37.855990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.855963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:37.918406 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tk9\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.918735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.918679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:37.966346 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:37.966312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:38.020060 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tk9\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.020926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.020867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.022516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.022497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.022882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.022866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.029945 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.029922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.030231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.030207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tk9\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9\") pod \"e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") " pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.119841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.119811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:38.399043 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.398956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:29:38.405535 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:29:38.405503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2a502e_7d67_4551_93ba_976db6ea5763.slice/crio-f9f74d1857d9dfa25b3c52188b44ec628040ef1c32d5777f3c272e4ea1a069f2 WatchSource:0}: Error finding container f9f74d1857d9dfa25b3c52188b44ec628040ef1c32d5777f3c272e4ea1a069f2: Status 404 returned error can't find the container with id f9f74d1857d9dfa25b3c52188b44ec628040ef1c32d5777f3c272e4ea1a069f2
Sep 10 11:29:38.407918 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.407883    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:38.408005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.407953    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:38.408005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.407981    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:29:38.703170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.703053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerStarted","Data":"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"}
Sep 10 11:29:38.703170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.703139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerStarted","Data":"f9f74d1857d9dfa25b3c52188b44ec628040ef1c32d5777f3c272e4ea1a069f2"}
Sep 10 11:29:38.732244 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:38.732187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podStartSLOduration=1.732168872 podStartE2EDuration="1.732168872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:29:38.730388727 +0000 UTC m=+896.805714392" watchObservedRunningTime="2026-09-10 11:29:38.732168872 +0000 UTC m=+896.807494607"
Sep 10 11:29:39.118431 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.118405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:39.120400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.120379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:39.125320 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.125299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:39.241815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.241781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:39.341541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.341512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:39.440590 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.440507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:39.708836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.708754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:39.709843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.709824    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:29:39.847069 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:39.847034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:41.025991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:41.025961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:41.175054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:41.175020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:42.269903 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.269873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:29:42.365134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365273    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zx97\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token\") pod \"3e65cb15-18fc-47cb-80ba-9c132702a099\" (UID: \"3e65cb15-18fc-47cb-80ba-9c132702a099\") "
Sep 10 11:29:42.365666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:42.365815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:29:42.365815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365758    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65cb15-18fc-47cb-80ba-9c132702a099-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.365815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365776    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.365981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:42.365981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:42.365981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.365934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:42.368174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.368147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97" (OuterVolumeSpecName: "kube-api-access-4zx97") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "kube-api-access-4zx97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:42.368307 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.368242    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:29:42.368534 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.368516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:29:42.368590 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.368528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e65cb15-18fc-47cb-80ba-9c132702a099" (UID: "3e65cb15-18fc-47cb-80ba-9c132702a099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:29:42.425278 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.425246    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:29:42.426259 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.426237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:29:42.466966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466935    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.466966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466963    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.466966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466971    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.467285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466979    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.467285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zx97\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-kube-api-access-4zx97\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.467285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.466995    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65cb15-18fc-47cb-80ba-9c132702a099-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.467285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.467003    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65cb15-18fc-47cb-80ba-9c132702a099-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:29:42.484991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.484949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:42.579772 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.579741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:42.726305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.726215    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerID="2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc" exitCode=0
Sep 10 11:29:42.726305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.726274    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"
Sep 10 11:29:42.726305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.726279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerDied","Data":"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"}
Sep 10 11:29:42.726543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.726334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj" event={"ID":"3e65cb15-18fc-47cb-80ba-9c132702a099","Type":"ContainerDied","Data":"97ade14a73a28b83fef449ed94b7b342e75491b2f89f51160999fa248a578e68"}
Sep 10 11:29:42.726543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.726361    2578 scope.go:117] "RemoveContainer" containerID="2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"
Sep 10 11:29:42.735057 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.735039    2578 scope.go:117] "RemoveContainer" containerID="0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"
Sep 10 11:29:42.742969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.742950    2578 scope.go:117] "RemoveContainer" containerID="2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"
Sep 10 11:29:42.743257 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:42.743238    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc\": container with ID starting with 2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc not found: ID does not exist" containerID="2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"
Sep 10 11:29:42.743313 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.743266    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc"} err="failed to get container status \"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc\": rpc error: code = NotFound desc = could not find container \"2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc\": container with ID starting with 2c50e1dbccc9d2021936ba1adf4c8d4028526e62d2639359beb0b3b8ba54e4cc not found: ID does not exist"
Sep 10 11:29:42.743313 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.743283    2578 scope.go:117] "RemoveContainer" containerID="0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"
Sep 10 11:29:42.743538 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:29:42.743522    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf\": container with ID starting with 0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf not found: ID does not exist" containerID="0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"
Sep 10 11:29:42.743579 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.743542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf"} err="failed to get container status \"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf\": rpc error: code = NotFound desc = could not find container \"0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf\": container with ID starting with 0191c89bd152184c6b2225cd74d9c2409f9b703d1c3f8d181ea69dd3f537b2cf not found: ID does not exist"
Sep 10 11:29:42.748827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.748807    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:29:42.756219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.755220    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8de26b-openshift-default-774f69876-tkckj"]
Sep 10 11:29:42.956856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:42.956814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:43.054466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.054383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:43.151273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.151245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:29:43.152505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152487    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:29:43.152843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152831    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:43.152882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca54ad06-d2f9-4663-bcdb-c74167ffee10" containerName="istio-proxy"
Sep 10 11:29:43.152882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.152882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152867    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.152882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152881    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.152991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152887    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.152991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152940    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.152991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.152947    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" containerName="istio-proxy"
Sep 10 11:29:43.157428 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.157414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.160317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.160295    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f39de1\""
Sep 10 11:29:43.161306 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.161292    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f39de1-dockercfg-btqqn\""
Sep 10 11:29:43.167778 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.167758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:29:43.274535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.274504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxqt\" (UniqueName: \"kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.274909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.274632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.375496 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.375462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxqt\" (UniqueName: \"kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.375703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.375561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.376249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.376229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.387954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.387917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxqt\" (UniqueName: \"kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt\") pod \"payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.468716 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.468682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:43.774064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:43.774003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:29:43.776205 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:29:43.776168    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9b405b_9329_4c7c_b1ac_5d83f42a09bf.slice/crio-aabf6b74e740fefc0ba7e5ca6690ffbf2b938a05c65cd3a76495255e9358acb6 WatchSource:0}: Error finding container aabf6b74e740fefc0ba7e5ca6690ffbf2b938a05c65cd3a76495255e9358acb6: Status 404 returned error can't find the container with id aabf6b74e740fefc0ba7e5ca6690ffbf2b938a05c65cd3a76495255e9358acb6
Sep 10 11:29:44.456007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:44.455973    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e65cb15-18fc-47cb-80ba-9c132702a099" path="/var/lib/kubelet/pods/3e65cb15-18fc-47cb-80ba-9c132702a099/volumes"
Sep 10 11:29:44.739915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:44.739823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" event={"ID":"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf","Type":"ContainerStarted","Data":"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"}
Sep 10 11:29:44.739915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:44.739877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" event={"ID":"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf","Type":"ContainerStarted","Data":"aabf6b74e740fefc0ba7e5ca6690ffbf2b938a05c65cd3a76495255e9358acb6"}
Sep 10 11:29:44.739915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:44.739906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:29:44.760645 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:44.760591    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" podStartSLOduration=1.760576764 podStartE2EDuration="1.760576764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:29:44.759069677 +0000 UTC m=+902.834395341" watchObservedRunningTime="2026-09-10 11:29:44.760576764 +0000 UTC m=+902.835902427"
Sep 10 11:29:55.747315 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:29:55.747276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:30:00.163282 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.163241    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"]
Sep 10 11:30:00.168162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.168139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"]
Sep 10 11:30:00.168288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.168266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"
Sep 10 11:30:00.171549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.171529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"
Sep 10 11:30:00.172454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.172428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:30:00.180311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.180218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"
Sep 10 11:30:00.181218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.181188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"]
Sep 10 11:30:00.191185 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.188101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"]
Sep 10 11:30:00.191185 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.191079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"
Sep 10 11:30:00.472065 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.472034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"]
Sep 10 11:30:00.475393 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:30:00.475369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5a1fb9_d824_4d81_8a5e_abb369cd6816.slice/crio-075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542 WatchSource:0}: Error finding container 075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542: Status 404 returned error can't find the container with id 075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542
Sep 10 11:30:00.496321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.496289    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"]
Sep 10 11:30:00.498349 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:30:00.498317    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bb9535_df52_4569_ac9d_0b61a2ba7eca.slice/crio-6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f WatchSource:0}: Error finding container 6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f: Status 404 returned error can't find the container with id 6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f
Sep 10 11:30:00.847769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.847715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54" event={"ID":"91bb9535-df52-4569-ac9d-0b61a2ba7eca","Type":"ContainerStarted","Data":"2448edacb8af52041ffa248b0d1b333722d531944d93876aaadffd44b8015300"}
Sep 10 11:30:00.847970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.847781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54" event={"ID":"91bb9535-df52-4569-ac9d-0b61a2ba7eca","Type":"ContainerStarted","Data":"6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f"}
Sep 10 11:30:00.849408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.849381    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerID="0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b" exitCode=0
Sep 10 11:30:00.849535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.849456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerDied","Data":"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"}
Sep 10 11:30:00.849846 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.849830    2578 scope.go:117] "RemoveContainer" containerID="0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"
Sep 10 11:30:00.851011 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.850973    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:00.851130 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.851016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww" event={"ID":"8a5a1fb9-d824-4d81-8a5e-abb369cd6816","Type":"ContainerStarted","Data":"0104c6efa4e68ab1e310bb294f06ec3f1e0530d267b8a12fcbae925568f2cfe3"}
Sep 10 11:30:00.851130 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.851057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww" event={"ID":"8a5a1fb9-d824-4d81-8a5e-abb369cd6816","Type":"ContainerStarted","Data":"075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542"}
Sep 10 11:30:00.851130 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.851103    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:00.851245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.851152    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:00.872852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.872764    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54" podStartSLOduration=0.872738806 podStartE2EDuration="872.738806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:30:00.870940448 +0000 UTC m=+918.946266113" watchObservedRunningTime="2026-09-10 11:30:00.872738806 +0000 UTC m=+918.948064473"
Sep 10 11:30:00.933605 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:00.933555    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww" podStartSLOduration=0.933539781 podStartE2EDuration="933.539781ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:30:00.931250887 +0000 UTC m=+919.006576552" watchObservedRunningTime="2026-09-10 11:30:00.933539781 +0000 UTC m=+919.008865444"
Sep 10 11:30:01.858823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:01.858792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerStarted","Data":"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"}
Sep 10 11:30:01.860388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:01.860362    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a5a1fb9-d824-4d81-8a5e-abb369cd6816" containerID="0104c6efa4e68ab1e310bb294f06ec3f1e0530d267b8a12fcbae925568f2cfe3" exitCode=0
Sep 10 11:30:01.860504 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:01.860438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww" event={"ID":"8a5a1fb9-d824-4d81-8a5e-abb369cd6816","Type":"ContainerDied","Data":"0104c6efa4e68ab1e310bb294f06ec3f1e0530d267b8a12fcbae925568f2cfe3"}
Sep 10 11:30:01.861907 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:01.861885    2578 generic.go:356] "Generic (PLEG): container finished" podID="91bb9535-df52-4569-ac9d-0b61a2ba7eca" containerID="2448edacb8af52041ffa248b0d1b333722d531944d93876aaadffd44b8015300" exitCode=0
Sep 10 11:30:01.861993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:01.861943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54" event={"ID":"91bb9535-df52-4569-ac9d-0b61a2ba7eca","Type":"ContainerDied","Data":"2448edacb8af52041ffa248b0d1b333722d531944d93876aaadffd44b8015300"}
Sep 10 11:30:02.120951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:02.120855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:02.121530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:02.121506    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:02.121592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:02.121555    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:03.051597 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.051574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"
Sep 10 11:30:03.068252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.068224    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"
Sep 10 11:30:03.120701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.120604    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:03.120701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.120669    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:03.873656 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.873625    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"
Sep 10 11:30:03.873845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.873634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww" event={"ID":"8a5a1fb9-d824-4d81-8a5e-abb369cd6816","Type":"ContainerDied","Data":"075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542"}
Sep 10 11:30:03.873845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.873748    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="075dc76c4bea06171b1e4d34e289b8afca47804f95010619c143160d298b8542"
Sep 10 11:30:03.875051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.875030    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"
Sep 10 11:30:03.875218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.875028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54" event={"ID":"91bb9535-df52-4569-ac9d-0b61a2ba7eca","Type":"ContainerDied","Data":"6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f"}
Sep 10 11:30:03.875218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:03.875148    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d38adedf3e18988015a28397e6c7bc1b2d80dbc48e594224124ccd4fc84fc3f"
Sep 10 11:30:04.120264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:04.120223    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:04.120755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:04.120285    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:04.407408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:04.407373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:05.120643 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:05.120607    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:05.121053 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:05.120664    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:05.822490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:05.822454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:05.922533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:05.922501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:06.120180 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:06.120143    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:06.120356 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:06.120197    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:07.121018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:07.120987    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 11:30:07.121447 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:07.121039    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 11:30:08.124862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:08.124832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:08.125246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:08.125121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:08.125712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:08.125693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:12.535314 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:12.535269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:12.768687 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:12.768652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:12.980152 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:12.980100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:14.756399 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:14.756798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756738    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91bb9535-df52-4569-ac9d-0b61a2ba7eca" containerName="cleanup"
Sep 10 11:30:14.756798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756749    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="91bb9535-df52-4569-ac9d-0b61a2ba7eca" containerName="cleanup"
Sep 10 11:30:14.756798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5a1fb9-d824-4d81-8a5e-abb369cd6816" containerName="cleanup"
Sep 10 11:30:14.756798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5a1fb9-d824-4d81-8a5e-abb369cd6816" containerName="cleanup"
Sep 10 11:30:14.756932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756825    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="91bb9535-df52-4569-ac9d-0b61a2ba7eca" containerName="cleanup"
Sep 10 11:30:14.756932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.756835    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5a1fb9-d824-4d81-8a5e-abb369cd6816" containerName="cleanup"
Sep 10 11:30:14.760012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.759992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.763862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.763837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f39de1\"/\"openshift-service-ca.crt\""
Sep 10 11:30:14.765375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.765342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f39de1\"/\"explicit-model-f39de1-kserve-self-signed-certs\""
Sep 10 11:30:14.765520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.765349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f39de1\"/\"default-dockercfg-9l9wm\""
Sep 10 11:30:14.765520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.765404    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f39de1\"/\"kube-root-ca.crt\""
Sep 10 11:30:14.774417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.774394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:14.802809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.802774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.802955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.802908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.803026 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.802978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rzz\" (UniqueName: \"kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.803026 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.803019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.803147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.803052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.803147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.803137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.807658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.807617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:14.908983 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.907338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909283 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909428 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.908926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909606 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909698 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rzz\" (UniqueName: \"kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909698 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.909865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.909816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.910139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.910118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.912965 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.912933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.913647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.913624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:14.923314 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:14.923281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rzz\" (UniqueName: \"kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz\") pod \"explicit-model-f39de1-kserve-6f878f797c-nlpmw\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") " pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:15.070486 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:15.070383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:15.353413 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:15.353385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:15.355706 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:30:15.355681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68fbe2c5_504e_4d4b_a73f_5d57beb9f750.slice/crio-47d0f7e81d7edc9b64a1053594870963d59eb6123468796401c8fd6d8eb9007a WatchSource:0}: Error finding container 47d0f7e81d7edc9b64a1053594870963d59eb6123468796401c8fd6d8eb9007a: Status 404 returned error can't find the container with id 47d0f7e81d7edc9b64a1053594870963d59eb6123468796401c8fd6d8eb9007a
Sep 10 11:30:15.944847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:15.944815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" event={"ID":"68fbe2c5-504e-4d4b-a73f-5d57beb9f750","Type":"ContainerStarted","Data":"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"}
Sep 10 11:30:15.945260 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:15.944860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" event={"ID":"68fbe2c5-504e-4d4b-a73f-5d57beb9f750","Type":"ContainerStarted","Data":"47d0f7e81d7edc9b64a1053594870963d59eb6123468796401c8fd6d8eb9007a"}
Sep 10 11:30:15.970608 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:15.970554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" podStartSLOduration=1.970536976 podStartE2EDuration="1.970536976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:30:15.969472616 +0000 UTC m=+934.044798281" watchObservedRunningTime="2026-09-10 11:30:15.970536976 +0000 UTC m=+934.045862666"
Sep 10 11:30:25.071393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:25.071351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:25.071830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:25.071550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:25.083802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:25.083781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:26.011180 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:26.011144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:27.131508 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:27.131472    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:27.131978 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:27.131568    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:27.631546503 +0000 UTC m=+945.706872148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:27.152285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.152257    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:27.186819 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.186757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:27.292562 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.292530    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"]
Sep 10 11:30:27.296209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.296190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:27.299264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.299242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:30:27.299384 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.299278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:30:27.306939 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.306901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"]
Sep 10 11:30:27.434852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.434741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f39de1-4fms5\" (UID: \"a0902107-b935-4cd2-aa48-5de66e749b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:27.536075 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.536032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f39de1-4fms5\" (UID: \"a0902107-b935-4cd2-aa48-5de66e749b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:27.536672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.536648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f39de1-4fms5\" (UID: \"a0902107-b935-4cd2-aa48-5de66e749b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:27.609070 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.609034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:27.637499 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:27.637470    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:27.637650 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:27.637550    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:28.637531484 +0000 UTC m=+946.712857136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:27.883322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:27.883293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"]
Sep 10 11:30:27.885164 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:30:27.885119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0902107_b935_4cd2_aa48_5de66e749b69.slice/crio-7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05 WatchSource:0}: Error finding container 7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05: Status 404 returned error can't find the container with id 7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05
Sep 10 11:30:28.012525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:28.012485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5" event={"ID":"a0902107-b935-4cd2-aa48-5de66e749b69","Type":"ContainerStarted","Data":"7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05"}
Sep 10 11:30:28.650202 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:28.650150    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:28.650750 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:28.650256    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:30.650230963 +0000 UTC m=+948.725556619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:28.822853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:28.822814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:29.019244 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:29.019152    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0902107-b935-4cd2-aa48-5de66e749b69" containerID="365ba81ef789c7ab10021e01790824a091d2cc82965774615b389b2ab645b119" exitCode=0
Sep 10 11:30:29.019397 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:29.019246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5" event={"ID":"a0902107-b935-4cd2-aa48-5de66e749b69","Type":"ContainerDied","Data":"365ba81ef789c7ab10021e01790824a091d2cc82965774615b389b2ab645b119"}
Sep 10 11:30:30.164476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:30.164446    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:30.266097 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:30.266062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca\") pod \"a0902107-b935-4cd2-aa48-5de66e749b69\" (UID: \"a0902107-b935-4cd2-aa48-5de66e749b69\") "
Sep 10 11:30:30.266462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:30.266435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0902107-b935-4cd2-aa48-5de66e749b69" (UID: "a0902107-b935-4cd2-aa48-5de66e749b69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:30:30.366962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:30.366923    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0902107-b935-4cd2-aa48-5de66e749b69-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:30.671250 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:30.671146    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:30.671250 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:30.671238    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:34.67121997 +0000 UTC m=+952.746545617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:31.031764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:31.031687    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"
Sep 10 11:30:31.031764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:31.031721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5" event={"ID":"a0902107-b935-4cd2-aa48-5de66e749b69","Type":"ContainerDied","Data":"7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05"}
Sep 10 11:30:31.031970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:31.031772    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c64e0b5ebea75a25b646836a52eeed591cbc175d3e8f8b5184cd46e9cb23c05"
Sep 10 11:30:34.711227 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:34.711186    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:34.711633 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:34.711265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:42.71124873 +0000 UTC m=+960.786574373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:35.555622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:35.555580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:35.664333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:35.664285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:36.027410 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:36.027381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:38.098963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.098922    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:30:38.099756 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.099693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" podUID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" containerName="payload-pre-processing" containerID="cri-o://312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece" gracePeriod=30
Sep 10 11:30:38.180816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.180781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:38.347008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.346973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"]
Sep 10 11:30:38.356132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.356100    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f39de1-4fms5"]
Sep 10 11:30:38.371512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.371480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:30:38.447179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.447137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume\") pod \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") "
Sep 10 11:30:38.447380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.447200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjxqt\" (UniqueName: \"kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt\") pod \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\" (UID: \"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf\") "
Sep 10 11:30:38.447603 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.447575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" (UID: "4c9b405b-9329-4c7c-b1ac-5d83f42a09bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:30:38.449417 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.449396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt" (OuterVolumeSpecName: "kube-api-access-vjxqt") pod "4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" (UID: "4c9b405b-9329-4c7c-b1ac-5d83f42a09bf"). InnerVolumeSpecName "kube-api-access-vjxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:30:38.455576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.455535    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0902107-b935-4cd2-aa48-5de66e749b69" path="/var/lib/kubelet/pods/a0902107-b935-4cd2-aa48-5de66e749b69/volumes"
Sep 10 11:30:38.548061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.548024    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:38.548061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:38.548056    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxqt\" (UniqueName: \"kubernetes.io/projected/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf-kube-api-access-vjxqt\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:39.078666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.078635    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" containerID="312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece" exitCode=0
Sep 10 11:30:39.078836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.078679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"
Sep 10 11:30:39.078836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.078713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" event={"ID":"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf","Type":"ContainerDied","Data":"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"}
Sep 10 11:30:39.078836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.078755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj" event={"ID":"4c9b405b-9329-4c7c-b1ac-5d83f42a09bf","Type":"ContainerDied","Data":"aabf6b74e740fefc0ba7e5ca6690ffbf2b938a05c65cd3a76495255e9358acb6"}
Sep 10 11:30:39.078836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.078784    2578 scope.go:117] "RemoveContainer" containerID="312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"
Sep 10 11:30:39.088468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.088442    2578 scope.go:117] "RemoveContainer" containerID="312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"
Sep 10 11:30:39.088803 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:39.088781    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece\": container with ID starting with 312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece not found: ID does not exist" containerID="312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"
Sep 10 11:30:39.088885 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.088816    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece"} err="failed to get container status \"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece\": rpc error: code = NotFound desc = could not find container \"312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece\": container with ID starting with 312d352a6abea72c2a40f07208ae282a1ccd52561f37f7e0f4bd55a88eb18ece not found: ID does not exist"
Sep 10 11:30:39.101191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.101161    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:30:39.107399 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.107367    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f39de1-84d7c74cdb-6b9gj"]
Sep 10 11:30:39.400377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.400343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:39.525951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:39.525922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:40.456419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:40.456388    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" path="/var/lib/kubelet/pods/4c9b405b-9329-4c7c-b1ac-5d83f42a09bf/volumes"
Sep 10 11:30:42.790384 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:42.790291    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-self-signed-certs: secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:42.790384 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:42.790366    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs podName:68fbe2c5-504e-4d4b-a73f-5d57beb9f750 nodeName:}" failed. No retries permitted until 2026-09-10 11:30:58.790349648 +0000 UTC m=+976.865675290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs") pod "explicit-model-f39de1-kserve-6f878f797c-nlpmw" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750") : secret "explicit-model-f39de1-kserve-self-signed-certs" not found
Sep 10 11:30:44.040313 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.040238    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" podUID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" containerName="main" containerID="cri-o://7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59" gracePeriod=45
Sep 10 11:30:44.301197 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.301132    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:44.407059 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6rzz\" (UniqueName: \"kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz\") pod \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\" (UID: \"68fbe2c5-504e-4d4b-a73f-5d57beb9f750\") "
Sep 10 11:30:44.407448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache" (OuterVolumeSpecName: "model-cache") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:44.407448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home" (OuterVolumeSpecName: "home") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:44.407603 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:44.407706 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407683    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.407841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407710    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.407841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.407723    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.409726 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.409696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:30:44.409807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.409732    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm" (OuterVolumeSpecName: "dshm") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:44.409842 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.409798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz" (OuterVolumeSpecName: "kube-api-access-p6rzz") pod "68fbe2c5-504e-4d4b-a73f-5d57beb9f750" (UID: "68fbe2c5-504e-4d4b-a73f-5d57beb9f750"). InnerVolumeSpecName "kube-api-access-p6rzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:30:44.509056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.509005    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.509056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.509040    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.509056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.509050    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rzz\" (UniqueName: \"kubernetes.io/projected/68fbe2c5-504e-4d4b-a73f-5d57beb9f750-kube-api-access-p6rzz\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:44.971199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:44.971167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:45.115296 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.115263    2578 generic.go:356] "Generic (PLEG): container finished" podID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" containerID="7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59" exitCode=0
Sep 10 11:30:45.115734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.115307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"
Sep 10 11:30:45.115734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.115347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" event={"ID":"68fbe2c5-504e-4d4b-a73f-5d57beb9f750","Type":"ContainerDied","Data":"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"}
Sep 10 11:30:45.115734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.115390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw" event={"ID":"68fbe2c5-504e-4d4b-a73f-5d57beb9f750","Type":"ContainerDied","Data":"47d0f7e81d7edc9b64a1053594870963d59eb6123468796401c8fd6d8eb9007a"}
Sep 10 11:30:45.115734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.115407    2578 scope.go:117] "RemoveContainer" containerID="7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"
Sep 10 11:30:45.124254 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.124238    2578 scope.go:117] "RemoveContainer" containerID="7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"
Sep 10 11:30:45.124517 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:45.124496    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59\": container with ID starting with 7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59 not found: ID does not exist" containerID="7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"
Sep 10 11:30:45.124592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.124524    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59"} err="failed to get container status \"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59\": rpc error: code = NotFound desc = could not find container \"7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59\": container with ID starting with 7abf25a8d7b8bb98d255ec9487c38d6b06a991be6bec535eba3dbebbb21a1f59 not found: ID does not exist"
Sep 10 11:30:45.134295 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.134272    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:45.139997 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:45.139966    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f39de1/explicit-model-f39de1-kserve-6f878f797c-nlpmw"]
Sep 10 11:30:46.288453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:46.288419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:46.455888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:46.455852    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" path="/var/lib/kubelet/pods/68fbe2c5-504e-4d4b-a73f-5d57beb9f750/volumes"
Sep 10 11:30:47.108846 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:47.108803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:49.836066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:49.836034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:49.915418 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:49.915387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:50.230443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:50.230414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:50.463677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:50.463641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:30:50.463898 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:50.463876    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" containerID="cri-o://da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f" gracePeriod=30
Sep 10 11:30:50.495673 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:50.495596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:51.009920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.009887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:51.278413 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278337    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:30:51.278737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278723    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" containerName="payload-pre-processing"
Sep 10 11:30:51.278791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278738    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" containerName="payload-pre-processing"
Sep 10 11:30:51.278791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278746    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0902107-b935-4cd2-aa48-5de66e749b69" containerName="revoke-keys"
Sep 10 11:30:51.278791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278752    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0902107-b935-4cd2-aa48-5de66e749b69" containerName="revoke-keys"
Sep 10 11:30:51.278791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278766    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" containerName="main"
Sep 10 11:30:51.278791 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" containerName="main"
Sep 10 11:30:51.278953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278850    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9b405b-9329-4c7c-b1ac-5d83f42a09bf" containerName="payload-pre-processing"
Sep 10 11:30:51.278953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278866    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fbe2c5-504e-4d4b-a73f-5d57beb9f750" containerName="main"
Sep 10 11:30:51.278953 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.278874    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0902107-b935-4cd2-aa48-5de66e749b69" containerName="revoke-keys"
Sep 10 11:30:51.283209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.283190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.286203 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.286181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-395207-openshift-default-dockercfg-jqzvl\""
Sep 10 11:30:51.297190 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.297168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:30:51.322992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.322963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:51.367419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz274\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.367969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.367846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469064 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz274\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.469762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.470003 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.469975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.470138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.470056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.471764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.471744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.471823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.471772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.480958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.480928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.481187 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.481165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz274\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274\") pod \"e2e-noten-395207-openshift-default-5d55cd987b-g6h28\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") " pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.593768 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.593689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:51.785674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.785642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:51.876929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.876898    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:30:51.877278 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:30:51.877255    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef98219_40e0_48a3_bc62_06bf3a060b5c.slice/crio-2286bf3621f82a21e87d20cd1d515aa22d105788e5f361ecf515905e1d1d1013 WatchSource:0}: Error finding container 2286bf3621f82a21e87d20cd1d515aa22d105788e5f361ecf515905e1d1d1013: Status 404 returned error can't find the container with id 2286bf3621f82a21e87d20cd1d515aa22d105788e5f361ecf515905e1d1d1013
Sep 10 11:30:51.879405 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.879370    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:51.879494 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.879438    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:51.879494 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:51.879466    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:30:52.155704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.155607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" event={"ID":"5ef98219-40e0-48a3-bc62-06bf3a060b5c","Type":"ContainerStarted","Data":"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"}
Sep 10 11:30:52.155704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.155665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" event={"ID":"5ef98219-40e0-48a3-bc62-06bf3a060b5c","Type":"ContainerStarted","Data":"2286bf3621f82a21e87d20cd1d515aa22d105788e5f361ecf515905e1d1d1013"}
Sep 10 11:30:52.189432 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.189371    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" podStartSLOduration=1.189347976 podStartE2EDuration="1.189347976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:30:52.185240984 +0000 UTC m=+970.260566650" watchObservedRunningTime="2026-09-10 11:30:52.189347976 +0000 UTC m=+970.264673640"
Sep 10 11:30:52.504922 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.504822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:52.594362 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.594323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:52.599598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.599570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:52.613200 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.613162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:52.758578 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:52.758482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:53.128176 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:53.128142    2578 patch_prober.go:28] interesting pod/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:30:53.128335 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:53.128195    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:30:53.161532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:53.161505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:53.162484 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:53.162464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:30:54.587724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:54.587693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:54.711453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:54.711421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:55.718178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.718153    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:55.811134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4tk9\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:55.811640 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket\") pod \"8d2a502e-7d67-4551-93ba-976db6ea5763\" (UID: \"8d2a502e-7d67-4551-93ba-976db6ea5763\") "
Sep 10 11:30:55.811747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:55.811747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:30:55.811857 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811697    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:55.811914 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811859    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:55.812078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.811924    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2a502e-7d67-4551-93ba-976db6ea5763-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.812078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.812018    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.812078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.812031    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.813483 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.813451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:30:55.813604 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.813495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:30:55.813604 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.813544    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9" (OuterVolumeSpecName: "kube-api-access-z4tk9") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "kube-api-access-z4tk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:30:55.813779 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.813763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d2a502e-7d67-4551-93ba-976db6ea5763" (UID: "8d2a502e-7d67-4551-93ba-976db6ea5763"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:30:55.913353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913313    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.913353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913348    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tk9\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-kube-api-access-z4tk9\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.913353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913359    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.913594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913369    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.913594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913378    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2a502e-7d67-4551-93ba-976db6ea5763-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.913594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.913385    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2a502e-7d67-4551-93ba-976db6ea5763-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:30:55.991048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:55.991009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:56.086905 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.086826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:30:56.186467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.186433    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerID="da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f" exitCode=0
Sep 10 11:30:56.186622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.186490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"
Sep 10 11:30:56.186622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.186520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerDied","Data":"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"}
Sep 10 11:30:56.186622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.186606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs" event={"ID":"8d2a502e-7d67-4551-93ba-976db6ea5763","Type":"ContainerDied","Data":"f9f74d1857d9dfa25b3c52188b44ec628040ef1c32d5777f3c272e4ea1a069f2"}
Sep 10 11:30:56.186741 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.186635    2578 scope.go:117] "RemoveContainer" containerID="da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"
Sep 10 11:30:56.196248 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.196231    2578 scope.go:117] "RemoveContainer" containerID="0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"
Sep 10 11:30:56.207529 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.207505    2578 scope.go:117] "RemoveContainer" containerID="da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"
Sep 10 11:30:56.207819 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:56.207800    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f\": container with ID starting with da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f not found: ID does not exist" containerID="da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"
Sep 10 11:30:56.207870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.207831    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f"} err="failed to get container status \"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f\": rpc error: code = NotFound desc = could not find container \"da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f\": container with ID starting with da4037cedcf0cf6b5519fb20804726c245208c6936b1ef3dfb21e7e6a3784d7f not found: ID does not exist"
Sep 10 11:30:56.207870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.207850    2578 scope.go:117] "RemoveContainer" containerID="0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"
Sep 10 11:30:56.208144 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:30:56.208127    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b\": container with ID starting with 0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b not found: ID does not exist" containerID="0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"
Sep 10 11:30:56.208200 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.208153    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b"} err="failed to get container status \"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b\": rpc error: code = NotFound desc = could not find container \"0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b\": container with ID starting with 0f1281e404bbadd11212dcedb369da6ea826a88388d96970ac9c9f5b4120c14b not found: ID does not exist"
Sep 10 11:30:56.214211 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.214190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:30:56.219801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.219780    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f39de1-openshift-default-6bf685d45b-x52gs"]
Sep 10 11:30:56.456394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:56.456348    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" path="/var/lib/kubelet/pods/8d2a502e-7d67-4551-93ba-976db6ea5763/volumes"
Sep 10 11:30:57.205017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:30:57.204982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:03.764853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:03.764774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:03.845562 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:03.845530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:04.464802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:04.464764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:10.905548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:10.905501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:11.023250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.023205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:11.159980 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.159869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:11.538998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.538926    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:31:11.539309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.539296    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:11.539366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.539311    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:11.539366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.539319    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:11.539366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.539324    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:11.539461 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.539393    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:11.541916 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.541897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.551751 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.551721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5cf411f1-openshift-default-dockercfg-pk97n\""
Sep 10 11:31:11.564028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.563996    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:31:11.659507 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.659472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:11.668009 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.667986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l599j\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668121 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.668394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.668271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769189 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l599j\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769189 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.769982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.770058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.769989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.770058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.770029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.771888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.771861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.772041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.772026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.798683 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.798617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l599j\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.806309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.806285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token\") pod \"e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") " pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.856739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.856705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:11.970032 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:11.970003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:12.089538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.089461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:12.157683 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.157652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:31:12.159394 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:31:12.159364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53ad816_e687_4cdc_8eb3_5578de542a0c.slice/crio-6c1133e51c84562bbb353ca1a11f422821f470ba9fed6d8aae40cf182b45fb2f WatchSource:0}: Error finding container 6c1133e51c84562bbb353ca1a11f422821f470ba9fed6d8aae40cf182b45fb2f: Status 404 returned error can't find the container with id 6c1133e51c84562bbb353ca1a11f422821f470ba9fed6d8aae40cf182b45fb2f
Sep 10 11:31:12.161690 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.161655    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:12.161787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.161736    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:12.161787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.161771    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:12.276065 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.276022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerStarted","Data":"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"}
Sep 10 11:31:12.276235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.276122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerStarted","Data":"6c1133e51c84562bbb353ca1a11f422821f470ba9fed6d8aae40cf182b45fb2f"}
Sep 10 11:31:12.307612 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.307541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podStartSLOduration=1.307520946 podStartE2EDuration="1.307520946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:31:12.30304537 +0000 UTC m=+990.378371035" watchObservedRunningTime="2026-09-10 11:31:12.307520946 +0000 UTC m=+990.382846611"
Sep 10 11:31:12.857713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:12.857674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:13.862325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:13.862292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:14.287234 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:14.287143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:14.288267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:14.288243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:15.261336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:15.261303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:31:15.261743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:15.261613    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" podUID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" containerName="istio-proxy" containerID="cri-o://5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec" gracePeriod=30
Sep 10 11:31:15.853033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:15.853001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:15.971682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:15.971623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:16.154735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.154703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:16.475652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.475571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:16.604147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.604114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:16.786113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.785993    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:16.787221 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.787192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2a502e-7d67-4551-93ba-976db6ea5763" containerName="istio-proxy"
Sep 10 11:31:16.790307 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.790271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.795574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.795546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5cf411f1-dockercfg-ngd5n\""
Sep 10 11:31:16.795969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.795560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5cf411f1\""
Sep 10 11:31:16.801737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.801194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:16.813563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.813529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbtg\" (UniqueName: \"kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.813715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.813699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.914297 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.914258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.914505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.914340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbtg\" (UniqueName: \"kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.915002 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.914977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:16.923979 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:16.923957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbtg\" (UniqueName: \"kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg\") pod \"payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") " pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:17.106127 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:17.106066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:17.384480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:17.384443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:17.385489 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:31:17.385464    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280fafc_28a6_4f12_945c_69e595338088.slice/crio-b7b8d5c46bcaa25151bfa4a393246e15df9b34d87e7696b0ca62a348cad900ec WatchSource:0}: Error finding container b7b8d5c46bcaa25151bfa4a393246e15df9b34d87e7696b0ca62a348cad900ec: Status 404 returned error can't find the container with id b7b8d5c46bcaa25151bfa4a393246e15df9b34d87e7696b0ca62a348cad900ec
Sep 10 11:31:18.313005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:18.312968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" event={"ID":"b280fafc-28a6-4f12-945c-69e595338088","Type":"ContainerStarted","Data":"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"}
Sep 10 11:31:18.313405 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:18.313030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" event={"ID":"b280fafc-28a6-4f12-945c-69e595338088","Type":"ContainerStarted","Data":"b7b8d5c46bcaa25151bfa4a393246e15df9b34d87e7696b0ca62a348cad900ec"}
Sep 10 11:31:18.313405 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:18.313150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:18.334141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:18.334071    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" podStartSLOduration=2.334055459 podStartE2EDuration="2.334055459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:31:18.331994936 +0000 UTC m=+996.407320603" watchObservedRunningTime="2026-09-10 11:31:18.334055459 +0000 UTC m=+996.409381122"
Sep 10 11:31:19.739674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:19.739638    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:31:19.762284 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:19.762245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:20.532868 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.532796    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:31:20.554857 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.554818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.554872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.554898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.554945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.554997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555334 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555334 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz274\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555334 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555491 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:20.555491 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket\") pod \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\" (UID: \"5ef98219-40e0-48a3-bc62-06bf3a060b5c\") "
Sep 10 11:31:20.555491 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:20.555491 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:31:20.556181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:20.556181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555840    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.556181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555859    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.556181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.555874    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.556999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.556969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:20.561480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.559100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:20.562993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.562943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:20.565291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.565257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:31:20.565626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.565554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274" (OuterVolumeSpecName: "kube-api-access-dz274") pod "5ef98219-40e0-48a3-bc62-06bf3a060b5c" (UID: "5ef98219-40e0-48a3-bc62-06bf3a060b5c"). InnerVolumeSpecName "kube-api-access-dz274". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:20.657241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657204    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.657241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657234    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.657241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657244    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.657241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657252    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef98219-40e0-48a3-bc62-06bf3a060b5c-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.657527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657261    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.657527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.657269    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz274\" (UniqueName: \"kubernetes.io/projected/5ef98219-40e0-48a3-bc62-06bf3a060b5c-kube-api-access-dz274\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:20.716675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:20.716637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:21.203244 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.203210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:21.332477 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.332440    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" containerID="5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec" exitCode=0
Sep 10 11:31:21.332681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.332489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"
Sep 10 11:31:21.332681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.332524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" event={"ID":"5ef98219-40e0-48a3-bc62-06bf3a060b5c","Type":"ContainerDied","Data":"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"}
Sep 10 11:31:21.332681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.332579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28" event={"ID":"5ef98219-40e0-48a3-bc62-06bf3a060b5c","Type":"ContainerDied","Data":"2286bf3621f82a21e87d20cd1d515aa22d105788e5f361ecf515905e1d1d1013"}
Sep 10 11:31:21.332681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.332602    2578 scope.go:117] "RemoveContainer" containerID="5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"
Sep 10 11:31:21.342354 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.342330    2578 scope.go:117] "RemoveContainer" containerID="5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"
Sep 10 11:31:21.342686 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:31:21.342666    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec\": container with ID starting with 5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec not found: ID does not exist" containerID="5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"
Sep 10 11:31:21.342756 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.342693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec"} err="failed to get container status \"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec\": rpc error: code = NotFound desc = could not find container \"5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec\": container with ID starting with 5864a09d1dccadb0155669c3a8553d36c2b89908bdbbd502d719956888403cec not found: ID does not exist"
Sep 10 11:31:21.361257 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.361225    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:31:21.365191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.365159    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-395207-openshift-default-5d55cd987b-g6h28"]
Sep 10 11:31:21.401174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:21.400996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:22.455524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:22.455490    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" path="/var/lib/kubelet/pods/5ef98219-40e0-48a3-bc62-06bf3a060b5c/volumes"
Sep 10 11:31:29.289052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.289015    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:29.289462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.289099    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:29.321577 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.321541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:29.382991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.382951    2578 generic.go:356] "Generic (PLEG): container finished" podID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerID="1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49" exitCode=0
Sep 10 11:31:29.383200 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.383021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerDied","Data":"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"}
Sep 10 11:31:29.383521 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.383504    2578 scope.go:117] "RemoveContainer" containerID="1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"
Sep 10 11:31:29.384806 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.384773    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:29.384883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.384865    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:29.384934 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:29.384918    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:31:30.393915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:30.393871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerStarted","Data":"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"}
Sep 10 11:31:30.856973 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:30.856930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:30.857195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:30.857174    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:30.857261 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:30.857223    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:31.857325 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:31.857286    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:31.857744 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:31.857342    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:32.067237 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.067197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"]
Sep 10 11:31:32.070400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.070372    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8355df-29817330-594ww"]
Sep 10 11:31:32.073206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.073177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:32.123755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.123667    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:31:32.124266 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.124227    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" podUID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" containerName="payload-pre-processing" containerID="cri-o://d468078a8531746b1f674abe446880afbc45b1ee64cb8e6f2052ef113c167aec" gracePeriod=30
Sep 10 11:31:32.420184 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.420120    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" containerID="d468078a8531746b1f674abe446880afbc45b1ee64cb8e6f2052ef113c167aec" exitCode=0
Sep 10 11:31:32.420630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.420506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" event={"ID":"2c462f9f-1de1-45ab-9165-2b79d9b2d52a","Type":"ContainerDied","Data":"d468078a8531746b1f674abe446880afbc45b1ee64cb8e6f2052ef113c167aec"}
Sep 10 11:31:32.421250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.421132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs" event={"ID":"2c462f9f-1de1-45ab-9165-2b79d9b2d52a","Type":"ContainerDied","Data":"d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7"}
Sep 10 11:31:32.421448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.421432    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33a5cb047376d2cebd6d79f9327ce305d063de4b1708ae0ed9d3ce8be6fb5d7"
Sep 10 11:31:32.428115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.428071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:31:32.456975 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.456939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5a1fb9-d824-4d81-8a5e-abb369cd6816" path="/var/lib/kubelet/pods/8a5a1fb9-d824-4d81-8a5e-abb369cd6816/volumes"
Sep 10 11:31:32.509115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.507494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w25l\" (UniqueName: \"kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l\") pod \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") "
Sep 10 11:31:32.509115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.507555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume\") pod \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\" (UID: \"2c462f9f-1de1-45ab-9165-2b79d9b2d52a\") "
Sep 10 11:31:32.509692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.509664    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c462f9f-1de1-45ab-9165-2b79d9b2d52a" (UID: "2c462f9f-1de1-45ab-9165-2b79d9b2d52a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:31:32.511583 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.511556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l" (OuterVolumeSpecName: "kube-api-access-4w25l") pod "2c462f9f-1de1-45ab-9165-2b79d9b2d52a" (UID: "2c462f9f-1de1-45ab-9165-2b79d9b2d52a"). InnerVolumeSpecName "kube-api-access-4w25l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:32.609112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.608684    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w25l\" (UniqueName: \"kubernetes.io/projected/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-kube-api-access-4w25l\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:32.609112 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.608720    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c462f9f-1de1-45ab-9165-2b79d9b2d52a-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:32.857564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.857533    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:32.857956 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:32.857591    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:33.425744 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.425712    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"
Sep 10 11:31:33.451802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.451766    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:31:33.455851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.455819    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8355df-75b4bcff4b-sfbjs"]
Sep 10 11:31:33.661505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.661465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:33.857876 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.857849    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:33.858368 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:33.857902    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:34.456027 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:34.455997    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" path="/var/lib/kubelet/pods/2c462f9f-1de1-45ab-9165-2b79d9b2d52a/volumes"
Sep 10 11:31:34.760782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:34.760677    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" podUID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" containerName="main" containerID="cri-o://a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49" gracePeriod=45
Sep 10 11:31:34.857350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:34.857311    2578 patch_prober.go:28] interesting pod/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 11:31:34.857530 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:34.857369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 11:31:35.046715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.046681    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229669    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg2nj\" (UniqueName: \"kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.229825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229757    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir\") pod \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\" (UID: \"fd926c82-dcce-48e2-9454-6a444fbb5ebe\") "
Sep 10 11:31:35.230310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.229921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:35.230310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.230116    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.230310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.230142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:35.230468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.230323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home" (OuterVolumeSpecName: "home") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:35.233008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.232928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm" (OuterVolumeSpecName: "dshm") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:35.233199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.233118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj" (OuterVolumeSpecName: "kube-api-access-dg2nj") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "kube-api-access-dg2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:35.233292 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.233218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd926c82-dcce-48e2-9454-6a444fbb5ebe" (UID: "fd926c82-dcce-48e2-9454-6a444fbb5ebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:31:35.331453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.331358    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.331453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.331394    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.331453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.331402    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.331453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.331411    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg2nj\" (UniqueName: \"kubernetes.io/projected/fd926c82-dcce-48e2-9454-6a444fbb5ebe-kube-api-access-dg2nj\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.331453 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.331419    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd926c82-dcce-48e2-9454-6a444fbb5ebe-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:35.437850 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.437814    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" containerID="a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49" exitCode=0
Sep 10 11:31:35.438022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.437863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"
Sep 10 11:31:35.438022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.437881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" event={"ID":"fd926c82-dcce-48e2-9454-6a444fbb5ebe","Type":"ContainerDied","Data":"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"}
Sep 10 11:31:35.438022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.437922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447" event={"ID":"fd926c82-dcce-48e2-9454-6a444fbb5ebe","Type":"ContainerDied","Data":"c27b4a37b04f9fc1b28ff8cfa2918bfdbbcf209e2549af331473a910d777cb83"}
Sep 10 11:31:35.438022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.437941    2578 scope.go:117] "RemoveContainer" containerID="a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"
Sep 10 11:31:35.453723 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.453695    2578 scope.go:117] "RemoveContainer" containerID="a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"
Sep 10 11:31:35.454035 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:31:35.454015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49\": container with ID starting with a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49 not found: ID does not exist" containerID="a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"
Sep 10 11:31:35.454137 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.454042    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49"} err="failed to get container status \"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49\": rpc error: code = NotFound desc = could not find container \"a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49\": container with ID starting with a83df4e910d676fe2107a0988faccdde1675905a344ba700a891915d5e642b49 not found: ID does not exist"
Sep 10 11:31:35.464562 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.464536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:31:35.468516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.468490    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8355df/test-model-8355df-kserve-8688d4d6dc-6m447"]
Sep 10 11:31:35.930535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.930491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:35.930993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.930967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:35.932146 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:35.932126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:31:36.456655 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:36.456617    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" path="/var/lib/kubelet/pods/fd926c82-dcce-48e2-9454-6a444fbb5ebe/volumes"
Sep 10 11:31:37.051149 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:37.051120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:39.089616 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.089578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:39.189877 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.189834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:39.343222 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.343137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:31:39.343425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.343400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy" containerID="cri-o://5310e05d07bcdb108b0ff27bb0bc409fb964bfd893e9a9b0612868ca9a5d2087" gracePeriod=30
Sep 10 11:31:39.420329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.420297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:39.635135 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.635106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:39.909056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:39.908976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:40.257626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:40.257547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:40.351624 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:40.351592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:40.725854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:40.725822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:44.493797 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.493768    2578 generic.go:356] "Generic (PLEG): container finished" podID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerID="5310e05d07bcdb108b0ff27bb0bc409fb964bfd893e9a9b0612868ca9a5d2087" exitCode=0
Sep 10 11:31:44.494252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.493850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerDied","Data":"5310e05d07bcdb108b0ff27bb0bc409fb964bfd893e9a9b0612868ca9a5d2087"}
Sep 10 11:31:44.494252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.493917    2578 scope.go:117] "RemoveContainer" containerID="b7b163e4acac6bb18df79ead40d24b062a2d0e38695a260a4bc1545ac90a9850"
Sep 10 11:31:44.608235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.608208    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:31:44.623235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxmbq\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket\") pod \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\" (UID: \"76b3aa34-1b86-4bdc-a6cf-06362bcb772f\") "
Sep 10 11:31:44.623543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:44.623805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data" (OuterVolumeSpecName: "istio-data") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:44.623805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623714    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:44.623911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623846    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.623911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623867    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.623911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:44.623911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:31:44.623911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.623880    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.625825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.625792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq" (OuterVolumeSpecName: "kube-api-access-qxmbq") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "kube-api-access-qxmbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:44.625948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.625824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token" (OuterVolumeSpecName: "istio-token") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:44.625948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.625819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:31:44.625948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.625867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76b3aa34-1b86-4bdc-a6cf-06362bcb772f" (UID: "76b3aa34-1b86-4bdc-a6cf-06362bcb772f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:31:44.725056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725018    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxmbq\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-kube-api-access-qxmbq\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.725056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725047    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.725056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725056    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.725056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725064    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.725359 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725074    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:44.725359 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:44.725081    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3aa34-1b86-4bdc-a6cf-06362bcb772f-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:45.506326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:45.506296    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"
Sep 10 11:31:45.506757 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:45.506323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx" event={"ID":"76b3aa34-1b86-4bdc-a6cf-06362bcb772f","Type":"ContainerDied","Data":"8f6114b6fde4aefb4bf8e793fd6ef161f71137763481a80614c2a1200eb0b676"}
Sep 10 11:31:45.506757 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:45.506410    2578 scope.go:117] "RemoveContainer" containerID="5310e05d07bcdb108b0ff27bb0bc409fb964bfd893e9a9b0612868ca9a5d2087"
Sep 10 11:31:45.533161 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:45.533130    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:31:45.539288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:45.539250    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8355df-openshift-default-65f448cd5b-wmfxx"]
Sep 10 11:31:46.455751 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:46.455718    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" path="/var/lib/kubelet/pods/76b3aa34-1b86-4bdc-a6cf-06362bcb772f/volumes"
Sep 10 11:31:46.769657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:46.769567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:46.874580 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:46.874544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:47.162182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162147    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"]
Sep 10 11:31:47.162585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162565    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162588    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162602    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" containerName="main"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" containerName="main"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162633    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" containerName="istio-proxy"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162642    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" containerName="istio-proxy"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" containerName="payload-pre-processing"
Sep 10 11:31:47.162677 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" containerName="payload-pre-processing"
Sep 10 11:31:47.163025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162750    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c462f9f-1de1-45ab-9165-2b79d9b2d52a" containerName="payload-pre-processing"
Sep 10 11:31:47.163025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef98219-40e0-48a3-bc62-06bf3a060b5c" containerName="istio-proxy"
Sep 10 11:31:47.163025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:47.163025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162793    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:47.163025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.162802    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd926c82-dcce-48e2-9454-6a444fbb5ebe" containerName="main"
Sep 10 11:31:47.167267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.167244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:47.171020 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.170997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:31:47.171171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.171068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:31:47.177944 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.177917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"]
Sep 10 11:31:47.247442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.247405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd\" (UID: \"0d10cf70-16cd-42a0-a537-9c10b049585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:47.348327 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.348291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd\" (UID: \"0d10cf70-16cd-42a0-a537-9c10b049585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:47.348881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.348862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd\" (UID: \"0d10cf70-16cd-42a0-a537-9c10b049585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:47.401235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.401200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:47.478701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.478620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:47.752745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:47.752722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"]
Sep 10 11:31:47.754597 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:31:47.754565    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d10cf70_16cd_42a0_a537_9c10b049585e.slice/crio-56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865 WatchSource:0}: Error finding container 56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865: Status 404 returned error can't find the container with id 56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865
Sep 10 11:31:48.525385 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:48.525352    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d10cf70-16cd-42a0-a537-9c10b049585e" containerID="ea480344a91532d6f300369d1028d0184f6a138f8925be132cf4924df9985b4c" exitCode=0
Sep 10 11:31:48.525789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:48.525442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd" event={"ID":"0d10cf70-16cd-42a0-a537-9c10b049585e","Type":"ContainerDied","Data":"ea480344a91532d6f300369d1028d0184f6a138f8925be132cf4924df9985b4c"}
Sep 10 11:31:48.525789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:48.525508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd" event={"ID":"0d10cf70-16cd-42a0-a537-9c10b049585e","Type":"ContainerStarted","Data":"56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865"}
Sep 10 11:31:49.669648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:49.669617    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:49.771072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:49.771031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca\") pod \"0d10cf70-16cd-42a0-a537-9c10b049585e\" (UID: \"0d10cf70-16cd-42a0-a537-9c10b049585e\") "
Sep 10 11:31:49.771396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:49.771375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d10cf70-16cd-42a0-a537-9c10b049585e" (UID: "0d10cf70-16cd-42a0-a537-9c10b049585e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:31:49.872408 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:49.872363    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d10cf70-16cd-42a0-a537-9c10b049585e-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:50.057578 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.057550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:50.536939 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.536907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd" event={"ID":"0d10cf70-16cd-42a0-a537-9c10b049585e","Type":"ContainerDied","Data":"56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865"}
Sep 10 11:31:50.536939 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.536927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"
Sep 10 11:31:50.537191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.536964    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56dfdf8700aca92db257a36fccf704059a91f2115b9baebd45019471b1d65865"
Sep 10 11:31:50.717286 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.717203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:50.739200 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.736905    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"]
Sep 10 11:31:50.742796 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.742757    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4c9990-29817330-28m54"]
Sep 10 11:31:50.807777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.807743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:50.890735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.890702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:50.971030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:50.970952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:51.044823 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:51.044787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:51.363704 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:51.363676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:52.455660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:52.455631    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91bb9535-df52-4569-ac9d-0b61a2ba7eca" path="/var/lib/kubelet/pods/91bb9535-df52-4569-ac9d-0b61a2ba7eca/volumes"
Sep 10 11:31:57.296420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:57.296376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:57.956524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:57.956493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:57.986773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:57.986719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:57.987016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:57.986982    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" podUID="b280fafc-28a6-4f12-945c-69e595338088" containerName="payload-processing" containerID="cri-o://732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460" gracePeriod=30
Sep 10 11:31:58.057558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.057529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.205729 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.205701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.223672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.223600    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"]
Sep 10 11:31:58.226686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.226661    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cf411f1-g85zd"]
Sep 10 11:31:58.244421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.244399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:58.285615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.285580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.346885 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.346843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume\") pod \"b280fafc-28a6-4f12-945c-69e595338088\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") "
Sep 10 11:31:58.347525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.346962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blbtg\" (UniqueName: \"kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg\") pod \"b280fafc-28a6-4f12-945c-69e595338088\" (UID: \"b280fafc-28a6-4f12-945c-69e595338088\") "
Sep 10 11:31:58.347525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.347261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b280fafc-28a6-4f12-945c-69e595338088" (UID: "b280fafc-28a6-4f12-945c-69e595338088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:31:58.349383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.349354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg" (OuterVolumeSpecName: "kube-api-access-blbtg") pod "b280fafc-28a6-4f12-945c-69e595338088" (UID: "b280fafc-28a6-4f12-945c-69e595338088"). InnerVolumeSpecName "kube-api-access-blbtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:31:58.448266 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.448210    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blbtg\" (UniqueName: \"kubernetes.io/projected/b280fafc-28a6-4f12-945c-69e595338088-kube-api-access-blbtg\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:58.448266 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.448257    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b280fafc-28a6-4f12-945c-69e595338088-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:31:58.456203 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.456175    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d10cf70-16cd-42a0-a537-9c10b049585e" path="/var/lib/kubelet/pods/0d10cf70-16cd-42a0-a537-9c10b049585e/volumes"
Sep 10 11:31:58.580258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.580174    2578 generic.go:356] "Generic (PLEG): container finished" podID="b280fafc-28a6-4f12-945c-69e595338088" containerID="732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460" exitCode=0
Sep 10 11:31:58.580258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.580228    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"
Sep 10 11:31:58.580258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.580241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" event={"ID":"b280fafc-28a6-4f12-945c-69e595338088","Type":"ContainerDied","Data":"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"}
Sep 10 11:31:58.580468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.580294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr" event={"ID":"b280fafc-28a6-4f12-945c-69e595338088","Type":"ContainerDied","Data":"b7b8d5c46bcaa25151bfa4a393246e15df9b34d87e7696b0ca62a348cad900ec"}
Sep 10 11:31:58.580468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.580317    2578 scope.go:117] "RemoveContainer" containerID="732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"
Sep 10 11:31:58.591114 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.591077    2578 scope.go:117] "RemoveContainer" containerID="732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"
Sep 10 11:31:58.591448 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:31:58.591424    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460\": container with ID starting with 732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460 not found: ID does not exist" containerID="732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"
Sep 10 11:31:58.591540 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.591461    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460"} err="failed to get container status \"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460\": rpc error: code = NotFound desc = could not find container \"732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460\": container with ID starting with 732f9b5c727652a5d0d4a6d047369d6cc6087d644ba1444d1fd53e7f10969460 not found: ID does not exist"
Sep 10 11:31:58.611701 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.611668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:58.628464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.628432    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cf411f1-c79bf9d6f-2jlcr"]
Sep 10 11:31:58.684657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.684623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.779304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.779266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.869820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.869785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:58.952847 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:58.952817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:59.045191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.045161    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:31:59.045416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.045391    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" containerID="cri-o://2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c" gracePeriod=30
Sep 10 11:31:59.080958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.080929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:59.654783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.654744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:31:59.678957 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.678928    2578 patch_prober.go:28] interesting pod/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:31:59.679120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.678985    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:31:59.967587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967497    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:31:59.967856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967842    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:59.967856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b3aa34-1b86-4bdc-a6cf-06362bcb772f" containerName="istio-proxy"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b280fafc-28a6-4f12-945c-69e595338088" containerName="payload-processing"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967892    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b280fafc-28a6-4f12-945c-69e595338088" containerName="payload-processing"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d10cf70-16cd-42a0-a537-9c10b049585e" containerName="revoke-keys"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d10cf70-16cd-42a0-a537-9c10b049585e" containerName="revoke-keys"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967951    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d10cf70-16cd-42a0-a537-9c10b049585e" containerName="revoke-keys"
Sep 10 11:31:59.968033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.967961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b280fafc-28a6-4f12-945c-69e595338088" containerName="payload-processing"
Sep 10 11:31:59.972131 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.972109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:31:59.975223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.975208    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5fde2a-openshift-default-dockercfg-tvvsg\""
Sep 10 11:31:59.985768 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:31:59.985744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:32:00.003962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.003934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:00.063077 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwm5\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063555 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.063597 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.063558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164605 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164605 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwm5\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.164880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.164987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.165065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.165221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165264 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.165256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.165359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.165603 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.165585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.167151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.167132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.167430 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.167411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.174431 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.174397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwm5\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.174547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.174502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token\") pod \"e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") " pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.207866 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.207831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:00.282169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.282046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:00.460989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.460944    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b280fafc-28a6-4f12-945c-69e595338088" path="/var/lib/kubelet/pods/b280fafc-28a6-4f12-945c-69e595338088/volumes"
Sep 10 11:32:00.568382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.568350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:32:00.570187 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:32:00.570163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c215354_a78c_4cee_988f_b632ce0b27d3.slice/crio-d9e6ba041e5bd3b935fd37c4409b1ee122988c948b6ca0528b39260232c52f11 WatchSource:0}: Error finding container d9e6ba041e5bd3b935fd37c4409b1ee122988c948b6ca0528b39260232c52f11: Status 404 returned error can't find the container with id d9e6ba041e5bd3b935fd37c4409b1ee122988c948b6ca0528b39260232c52f11
Sep 10 11:32:00.572835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.572801    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:00.572923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.572867    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:00.572923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.572896    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:00.594151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:00.594122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerStarted","Data":"d9e6ba041e5bd3b935fd37c4409b1ee122988c948b6ca0528b39260232c52f11"}
Sep 10 11:32:01.601328 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:01.601292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerStarted","Data":"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"}
Sep 10 11:32:01.631890 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:01.631829    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podStartSLOduration=2.631813867 podStartE2EDuration="2.631813867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:01.630817836 +0000 UTC m=+1039.706143501" watchObservedRunningTime="2026-09-10 11:32:01.631813867 +0000 UTC m=+1039.707139531"
Sep 10 11:32:02.283195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:02.283156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:02.288881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:02.288852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:02.606468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:02.606439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:02.607447 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:02.607425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:03.718591 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:03.718553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:03.925215 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:03.925183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:04.297896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.297868    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:32:04.403804 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.403974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zctr6\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket\") pod \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\" (UID: \"0fbfb02a-d608-46c2-ad08-b2349c55db5d\") "
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:04.404308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:04.404625 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404519    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.404688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:32:04.404742 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404707    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.404742 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404722    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.404843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.404806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:04.406469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.406444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:32:04.406469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.406463    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6" (OuterVolumeSpecName: "kube-api-access-zctr6") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "kube-api-access-zctr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:32:04.406707 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.406688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:32:04.406780 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.406748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fbfb02a-d608-46c2-ad08-b2349c55db5d" (UID: "0fbfb02a-d608-46c2-ad08-b2349c55db5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:04.505434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505400    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.505434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505428    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.505623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505441    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zctr6\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-kube-api-access-zctr6\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.505623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505455    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfb02a-d608-46c2-ad08-b2349c55db5d-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.505623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505464    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.505623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.505471    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfb02a-d608-46c2-ad08-b2349c55db5d-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:04.619017 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.618986    2578 generic.go:356] "Generic (PLEG): container finished" podID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerID="2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c" exitCode=0
Sep 10 11:32:04.619233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.619047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"
Sep 10 11:32:04.619233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.619076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerDied","Data":"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"}
Sep 10 11:32:04.619233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.619147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6" event={"ID":"0fbfb02a-d608-46c2-ad08-b2349c55db5d","Type":"ContainerDied","Data":"a0cad19258b60e1f0483d83da7a7d626cf6f7a1c004060cf9cb0d495931689e4"}
Sep 10 11:32:04.619233 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.619174    2578 scope.go:117] "RemoveContainer" containerID="2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"
Sep 10 11:32:04.628394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.628371    2578 scope.go:117] "RemoveContainer" containerID="9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"
Sep 10 11:32:04.636709 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.636690    2578 scope.go:117] "RemoveContainer" containerID="2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"
Sep 10 11:32:04.637008 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:32:04.636971    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c\": container with ID starting with 2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c not found: ID does not exist" containerID="2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"
Sep 10 11:32:04.637111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.637015    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c"} err="failed to get container status \"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c\": rpc error: code = NotFound desc = could not find container \"2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c\": container with ID starting with 2a3ffff11a54c2b6cca804b5d8d50eecc225c9eda3ea6bd174afe2fa2983c47c not found: ID does not exist"
Sep 10 11:32:04.637111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.637032    2578 scope.go:117] "RemoveContainer" containerID="9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"
Sep 10 11:32:04.637277 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:32:04.637263    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c\": container with ID starting with 9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c not found: ID does not exist" containerID="9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"
Sep 10 11:32:04.637318 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.637281    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c"} err="failed to get container status \"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c\": rpc error: code = NotFound desc = could not find container \"9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c\": container with ID starting with 9b7eedd528094bfb59d3a6b5bb7308c872ae8c6207b39e3edfe256fa9e43cc7c not found: ID does not exist"
Sep 10 11:32:04.645505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.645482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:32:04.658020 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:04.652937    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9990-openshift-default-59fb697cb4-5h9g6"]
Sep 10 11:32:05.014322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.014212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:05.105460 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.105427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:05.206433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.206400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:05.574504 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.574469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:05.672594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.672556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:05.851051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.850971    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:32:05.851462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.851442    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:05.851560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.851464    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:05.851560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.851475    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:05.851560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.851483    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:05.851709 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.851564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:05.855833 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.855810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:05.859344 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.859328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5fde2a\""
Sep 10 11:32:05.859463 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.859446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5fde2a-dockercfg-hm54k\""
Sep 10 11:32:05.867065 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.867040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:32:05.919110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.919051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:05.919305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:05.919184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqs2q\" (UniqueName: \"kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.020587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.020547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.021058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.020625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqs2q\" (UniqueName: \"kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.021326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.021307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.032747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.032713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqs2q\" (UniqueName: \"kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q\") pod \"payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.166655 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.166620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.380116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.380058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:06.446780 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:32:06.446744    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b3251_99be_4684_b1b0_99643174fae0.slice/crio-9d804f0cfb554c460b81cc5c1d3b13a8593e3f834a33bb5b2485fd9f24641687 WatchSource:0}: Error finding container 9d804f0cfb554c460b81cc5c1d3b13a8593e3f834a33bb5b2485fd9f24641687: Status 404 returned error can't find the container with id 9d804f0cfb554c460b81cc5c1d3b13a8593e3f834a33bb5b2485fd9f24641687
Sep 10 11:32:06.447329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.447223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:32:06.455619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.455595    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" path="/var/lib/kubelet/pods/0fbfb02a-d608-46c2-ad08-b2349c55db5d/volumes"
Sep 10 11:32:06.483821 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.483799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:06.633152 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.633115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" event={"ID":"393b3251-99be-4684-b1b0-99643174fae0","Type":"ContainerStarted","Data":"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"}
Sep 10 11:32:06.633353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.633174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" event={"ID":"393b3251-99be-4684-b1b0-99643174fae0","Type":"ContainerStarted","Data":"9d804f0cfb554c460b81cc5c1d3b13a8593e3f834a33bb5b2485fd9f24641687"}
Sep 10 11:32:06.633353 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.633218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:06.633492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.633371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:32:06.633594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.633571    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy" containerID="cri-o://dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb" gracePeriod=30
Sep 10 11:32:06.657443 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.657383    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" podStartSLOduration=1.657368028 podStartE2EDuration="1.657368028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:06.655344587 +0000 UTC m=+1044.730670252" watchObservedRunningTime="2026-09-10 11:32:06.657368028 +0000 UTC m=+1044.732693692"
Sep 10 11:32:06.666434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.666402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:06.761822 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:06.761738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:07.063195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:07.063109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:07.417006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:07.416966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:07.523433 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:07.523397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:11.931157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:11.931131    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:32:12.076899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.076866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.076899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.076906    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.076949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077178 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:12.077389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l599j\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data\") pod \"d53ad816-e687-4cdc-8eb3-5578de542a0c\" (UID: \"d53ad816-e687-4cdc-8eb3-5578de542a0c\") "
Sep 10 11:32:12.077389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:12.077626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:12.077626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:32:12.077785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:12.077862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077645    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.077908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077876    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.077908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.077894    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.079361 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.079340    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:32:12.079680 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.079655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j" (OuterVolumeSpecName: "kube-api-access-l599j") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "kube-api-access-l599j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:32:12.079850 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.079833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:32:12.079909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.079888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d53ad816-e687-4cdc-8eb3-5578de542a0c" (UID: "d53ad816-e687-4cdc-8eb3-5578de542a0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178645    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178677    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178687    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178695    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d53ad816-e687-4cdc-8eb3-5578de542a0c-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178704    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l599j\" (UniqueName: \"kubernetes.io/projected/d53ad816-e687-4cdc-8eb3-5578de542a0c-kube-api-access-l599j\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.178735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.178712    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d53ad816-e687-4cdc-8eb3-5578de542a0c-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:32:12.668665 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.668628    2578 generic.go:356] "Generic (PLEG): container finished" podID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerID="dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb" exitCode=0
Sep 10 11:32:12.668859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.668674    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"
Sep 10 11:32:12.668859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.668729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerDied","Data":"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"}
Sep 10 11:32:12.668859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.668778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8" event={"ID":"d53ad816-e687-4cdc-8eb3-5578de542a0c","Type":"ContainerDied","Data":"6c1133e51c84562bbb353ca1a11f422821f470ba9fed6d8aae40cf182b45fb2f"}
Sep 10 11:32:12.668859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.668801    2578 scope.go:117] "RemoveContainer" containerID="dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"
Sep 10 11:32:12.677625 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.677605    2578 scope.go:117] "RemoveContainer" containerID="1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"
Sep 10 11:32:12.687538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.687494    2578 scope.go:117] "RemoveContainer" containerID="dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"
Sep 10 11:32:12.688064 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:32:12.688024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb\": container with ID starting with dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb not found: ID does not exist" containerID="dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"
Sep 10 11:32:12.688213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.688066    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb"} err="failed to get container status \"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb\": rpc error: code = NotFound desc = could not find container \"dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb\": container with ID starting with dba9bbfd58337d478e5ce8702b8c5a52c32c9dc9c5f981bbaac2a234049fbdcb not found: ID does not exist"
Sep 10 11:32:12.688213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.688111    2578 scope.go:117] "RemoveContainer" containerID="1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"
Sep 10 11:32:12.688640 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:32:12.688598    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49\": container with ID starting with 1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49 not found: ID does not exist" containerID="1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"
Sep 10 11:32:12.688777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.688635    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49"} err="failed to get container status \"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49\": rpc error: code = NotFound desc = could not find container \"1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49\": container with ID starting with 1d9ff53dd0d51d421a2486d845f1dac74257f3c2b84565d4c8fce29e5f376f49 not found: ID does not exist"
Sep 10 11:32:12.698474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.698442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:32:12.714699 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:12.714576    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cf411f1-openshift-default-7db87856f6-zj9l8"]
Sep 10 11:32:13.605093 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:13.605019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:13.712912 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:13.712840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:13.808041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:13.808005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:13.976438 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:13.976351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:14.456671 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:14.456631    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" path="/var/lib/kubelet/pods/d53ad816-e687-4cdc-8eb3-5578de542a0c/volumes"
Sep 10 11:32:16.692728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.692643    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerID="b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee" exitCode=0
Sep 10 11:32:16.692728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.692704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerDied","Data":"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"}
Sep 10 11:32:16.693204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.693120    2578 scope.go:117] "RemoveContainer" containerID="b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"
Sep 10 11:32:16.694110 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.694062    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:16.694177 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.694152    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:16.694213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:16.694185    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:17.606899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:17.606800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:17.642880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:17.642853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:32:17.713290 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:17.713257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerStarted","Data":"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"}
Sep 10 11:32:18.282319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:18.282283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:18.283167 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:18.283141    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 11:32:18.283303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:18.283200    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 11:32:19.282573 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:19.282540    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 11:32:19.282958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:19.282592    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 11:32:20.282969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:20.282882    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 11:32:20.282969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:20.282935    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 11:32:20.469366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:20.469331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:20.549982 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:20.549889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:21.282732 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.282692    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 11:32:21.282899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.282757    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 11:32:21.449874 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.449837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:21.835298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835249    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:32:21.835594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:21.835594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835594    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:21.835737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835613    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:21.835737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835620    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:21.835737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835678    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:21.835737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.835688    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbfb02a-d608-46c2-ad08-b2349c55db5d" containerName="istio-proxy"
Sep 10 11:32:21.838741 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.838723    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.841964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.841947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c2e44c-openshift-default-dockercfg-j5v4t\""
Sep 10 11:32:21.863196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.863169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:32:21.915229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.914886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkb9k\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.996978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:21.999113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:21.997012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.098354 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.098245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.098646 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.098624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.099216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.099193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.099763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.099743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.100343 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.100512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.100688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.100803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.100930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkb9k\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101069 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.101610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.099683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.101497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.101679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.100716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.104338 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.104309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.125572 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.125540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.128160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.128135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkb9k\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k\") pod \"e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") " pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.149066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.149019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:22.282854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.282812    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 11:32:22.283033 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.282869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 11:32:22.367628 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.367570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:22.492289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.492244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:32:22.503122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.502375    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:22.503122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.502467    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:22.503122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.502516    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:22.742584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.742497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerStarted","Data":"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"}
Sep 10 11:32:22.742584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.742562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerStarted","Data":"dfabb65b8365000b437a13f5aef7152585ea8c1edd4090e18ca52866def860f1"}
Sep 10 11:32:22.774552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:22.774496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podStartSLOduration=1.774473901 podStartE2EDuration="1.774473901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:22.771049988 +0000 UTC m=+1060.846375654" watchObservedRunningTime="2026-09-10 11:32:22.774473901 +0000 UTC m=+1060.849799566"
Sep 10 11:32:23.149541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.149509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:23.154419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.154392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:23.286312 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.286281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:23.286609 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.286584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:23.287343 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.287324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:32:23.747740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.747703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:23.748654 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:23.748639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:24.731154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:24.731116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:24.824790 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:24.824757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:26.184789 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:26.184750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:26.284119 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:26.284067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:26.442380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:26.442292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:26.652487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:26.652452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:26.733271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:26.733186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:27.111830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:27.111803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:27.984104 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:27.984057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:28.645996 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:28.645966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:28.890162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:28.890124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:29.048963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.048874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:29.407174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.407142    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:32:29.407543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.407531    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ad816-e687-4cdc-8eb3-5578de542a0c" containerName="istio-proxy"
Sep 10 11:32:29.410674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.410658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.419802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.419779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-13c392-openshift-default-dockercfg-4dhmj\""
Sep 10 11:32:29.436712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.436682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:32:29.476873 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.476998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmsq\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.477033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.477065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.477385 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.477146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.497309 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.497268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:29.578252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmsq\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578894 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.578984 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.578963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.579029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.579015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.579111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.579045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.579253 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.579236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.580748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.580726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.580963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.580945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.587830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.587806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.587923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.587908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmsq\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq\") pod \"e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") " pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.590766 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.590720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:29.721930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.721838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:29.999988 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:29.999964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:32:30.002123 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:32:30.002074    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5992ce9_2d91_4d35_96a1_3e461b8f2f20.slice/crio-210cfd55a5b82d8a8d534a7846a856fbacc9b16e59185d26f22d8aa8516f97d1 WatchSource:0}: Error finding container 210cfd55a5b82d8a8d534a7846a856fbacc9b16e59185d26f22d8aa8516f97d1: Status 404 returned error can't find the container with id 210cfd55a5b82d8a8d534a7846a856fbacc9b16e59185d26f22d8aa8516f97d1
Sep 10 11:32:30.004450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.004418    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:30.004527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.004487    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:30.004527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.004513    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:30.791684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.791645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerStarted","Data":"75fedc94ae7674198a41a35c2008342f72817dd1ce81492fa45934ff1d55390b"}
Sep 10 11:32:30.792213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.791696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerStarted","Data":"210cfd55a5b82d8a8d534a7846a856fbacc9b16e59185d26f22d8aa8516f97d1"}
Sep 10 11:32:30.816428 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:30.816378    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podStartSLOduration=1.8163641190000002 podStartE2EDuration="1.816364119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:30.815035568 +0000 UTC m=+1068.890361237" watchObservedRunningTime="2026-09-10 11:32:30.816364119 +0000 UTC m=+1068.891689782"
Sep 10 11:32:31.722820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:31.722779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:31.728684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:31.728653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:31.797172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:31.797144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:31.798232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:31.798211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:32.950941 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:32.950906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:33.046742 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:33.046703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.011384 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.011326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.112493 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.112450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.211686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.211655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.574517 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.574485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.661567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.661516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:34.750029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.749995    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:32:34.753527 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.753510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.757216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.757192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-13c392\""
Sep 10 11:32:34.757329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.757228    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-13c392-dockercfg-ffvzb\""
Sep 10 11:32:34.766594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.766568    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:32:34.834812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.834700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.835185 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.835153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qsp\" (UniqueName: \"kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.936150 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.936064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qsp\" (UniqueName: \"kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.936306 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.936211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.936860 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.936841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:34.946396 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:34.946365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qsp\" (UniqueName: \"kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp\") pod \"payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:35.064657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.064622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:35.366100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.366057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:32:35.368074 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:32:35.368045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647cc619_9dfd_4913_9f78_368bc5de76e9.slice/crio-8be027c25bd7cc06c9e88eb4126c6691cb644095ab1fa91ec8ddfc311629fd7f WatchSource:0}: Error finding container 8be027c25bd7cc06c9e88eb4126c6691cb644095ab1fa91ec8ddfc311629fd7f: Status 404 returned error can't find the container with id 8be027c25bd7cc06c9e88eb4126c6691cb644095ab1fa91ec8ddfc311629fd7f
Sep 10 11:32:35.829510 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.829407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" event={"ID":"647cc619-9dfd-4913-9f78-368bc5de76e9","Type":"ContainerStarted","Data":"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"}
Sep 10 11:32:35.829510 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.829478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" event={"ID":"647cc619-9dfd-4913-9f78-368bc5de76e9","Type":"ContainerStarted","Data":"8be027c25bd7cc06c9e88eb4126c6691cb644095ab1fa91ec8ddfc311629fd7f"}
Sep 10 11:32:35.829695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.829593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:35.864944 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:35.864889    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" podStartSLOduration=1.864871324 podStartE2EDuration="1.864871324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:35.86207969 +0000 UTC m=+1073.937405353" watchObservedRunningTime="2026-09-10 11:32:35.864871324 +0000 UTC m=+1073.940197246"
Sep 10 11:32:37.586049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:37.586017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:39.789609 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:39.789254    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 11:32:39.789609 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:39.789326    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 11:32:41.866740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.866707    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerID="b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8" exitCode=0
Sep 10 11:32:41.867172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.866751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerDied","Data":"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"}
Sep 10 11:32:41.867172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.867166    2578 scope.go:117] "RemoveContainer" containerID="b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"
Sep 10 11:32:41.868278 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.868251    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:41.868346 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.868314    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:41.868383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:41.868343    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:42.873996 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:42.873962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerStarted","Data":"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"}
Sep 10 11:32:43.149733 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:43.149637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:43.149894 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:43.149746    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 11:32:43.149894 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:43.149793    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 11:32:44.149565 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:44.149533    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 11:32:44.149971 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:44.149585    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 11:32:45.149690 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:45.149657    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 11:32:45.150128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:45.149716    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 11:32:46.150514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:46.150464    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 11:32:46.150962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:46.150537    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 11:32:46.837210 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:46.837176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:32:47.110558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:47.110519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:47.150376 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:47.150344    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 11:32:47.150538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:47.150408    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 11:32:47.839321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:47.839268    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 11:32:47.839517 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:47.839339    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 11:32:48.167830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.167792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:48.168361 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.168208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:48.169498 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.169470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:32:48.919263 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.919233    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerID="75fedc94ae7674198a41a35c2008342f72817dd1ce81492fa45934ff1d55390b" exitCode=0
Sep 10 11:32:48.919480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.919315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerDied","Data":"75fedc94ae7674198a41a35c2008342f72817dd1ce81492fa45934ff1d55390b"}
Sep 10 11:32:48.919803 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.919789    2578 scope.go:117] "RemoveContainer" containerID="75fedc94ae7674198a41a35c2008342f72817dd1ce81492fa45934ff1d55390b"
Sep 10 11:32:48.920813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.920784    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:48.920892 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.920873    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:48.920929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:48.920918    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:49.747346 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:49.747258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:49.926909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:49.926873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerStarted","Data":"b3a9c2b50125c1a8261dc3fe9785c45a686cc7fe85ccff7d23397324479c1716"}
Sep 10 11:32:50.085465 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.085331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:50.401129 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.399904    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:32:50.416327 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.416293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.419858 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.419828    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-282355-openshift-default-dockercfg-f5t5k\""
Sep 10 11:32:50.432199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.432175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:32:50.434341 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.434308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:50.509593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.509782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.510123 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.510123 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.509863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v95\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v95\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.611740 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.611718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.612034 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.612017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.612427 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.612397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.612906 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.612880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.613199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.612968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.613199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.613016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.614505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.614478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.614655 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.614633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.627964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.627934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.628109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.628009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v95\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95\") pod \"e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") " pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:50.684642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.684529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:50.722771 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.722693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:50.723071 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.723013    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 10 11:32:50.723229 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.723076    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 10 11:32:50.726423 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:50.726377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:51.037038 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.037010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:32:51.038975 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:32:51.038946    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739e6cb2_8d8a_46c5_8418_48987394ea10.slice/crio-a6795b186f70aa2f3d7dfc0c53052305955d63fc4af49fe03c17e1f6e2351d3c WatchSource:0}: Error finding container a6795b186f70aa2f3d7dfc0c53052305955d63fc4af49fe03c17e1f6e2351d3c: Status 404 returned error can't find the container with id a6795b186f70aa2f3d7dfc0c53052305955d63fc4af49fe03c17e1f6e2351d3c
Sep 10 11:32:51.041231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.041200    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:51.041289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.041261    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:51.041336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.041292    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:32:51.722684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.722639    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 10 11:32:51.722883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.722706    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 10 11:32:51.940168 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.940133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerStarted","Data":"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"}
Sep 10 11:32:51.940345 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.940184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerStarted","Data":"a6795b186f70aa2f3d7dfc0c53052305955d63fc4af49fe03c17e1f6e2351d3c"}
Sep 10 11:32:51.964910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:51.964861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podStartSLOduration=1.96484508 podStartE2EDuration="1.96484508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:32:51.964255972 +0000 UTC m=+1090.039581637" watchObservedRunningTime="2026-09-10 11:32:51.96484508 +0000 UTC m=+1090.040170744"
Sep 10 11:32:52.722644 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.722611    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 10 11:32:52.723013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.722667    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 10 11:32:52.727808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.727791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:52.732761 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.732739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:52.945711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.945683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:52.946626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:52.946608    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:32:53.722618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:53.722582    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 10 11:32:53.722805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:53.722640    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 10 11:32:53.952774 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:53.952740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:54.060054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:54.059975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:54.722822 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:54.722785    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 10 11:32:54.723254 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:54.722848    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 10 11:32:54.995072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:54.994973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:55.812992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:55.812962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:55.813426 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:55.813407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:55.814196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:55.814173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:32:55.825862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:55.825839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:55.924961 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:55.924923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:56.155703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:56.155667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:56.461655 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:56.461570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:56.562899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:56.562867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:56.899716 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:56.899684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:57.719891 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:57.719860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:32:57.819849 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:32:57.819816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:12.059175 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.059143    2578 generic.go:356] "Generic (PLEG): container finished" podID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerID="46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953" exitCode=0
Sep 10 11:33:12.059598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.059188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerDied","Data":"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"}
Sep 10 11:33:12.059675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.059643    2578 scope.go:117] "RemoveContainer" containerID="46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"
Sep 10 11:33:12.060889 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.060860    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:12.060968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.060949    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:12.061013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:12.060999    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:13.068156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:13.068116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerStarted","Data":"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"}
Sep 10 11:33:13.308111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:13.307724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:13.726816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:13.726770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:33:13.727162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:13.727136    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 11:33:13.727223 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:13.727198    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 11:33:14.727025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:14.726985    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 11:33:14.727436 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:14.727054    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 11:33:15.727663 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:15.727625    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 11:33:15.728159 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:15.727742    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 11:33:16.727612 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:16.727573    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 11:33:16.727825 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:16.727684    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 11:33:17.727833 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:17.727795    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 11:33:17.728480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:17.728428    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 11:33:18.731387 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:18.731351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:33:18.731853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:18.731826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:33:18.732429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:18.732407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:33:18.974153 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:18.974121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:32.563372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.563341    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:33:32.567570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.567554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.570819 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.570799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"openshift-service-ca.crt\""
Sep 10 11:33:32.570932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.570894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"default-dockercfg-7wps5\""
Sep 10 11:33:32.572289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.572271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"kube-root-ca.crt\""
Sep 10 11:33:32.572596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.572582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"auth-test-model-13c392-kserve-self-signed-certs\""
Sep 10 11:33:32.588970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.588939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:33:32.621855 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.621823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:32.713483 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.713724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.713724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.713724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.713724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrqc\" (UniqueName: \"kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.713724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.713676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814351 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814351 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrqc\" (UniqueName: \"kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814351 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814907 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.814966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.814909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.816801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.816780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.816928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.816911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.824784 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.824761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrqc\" (UniqueName: \"kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc\") pod \"auth-test-model-13c392-kserve-57474bf699-v8qgq\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") " pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:32.876824 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:32.876765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:33.064797 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.064702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:33.159807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.159782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:33:33.162248 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:33:33.162216    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9569d44c_5aa3_4ea0_b768_77bba04b7acb.slice/crio-9d6fb2a91aacd8576ac94829dd54b18c9485f467ed31c5844e90b39f08c7c0ec WatchSource:0}: Error finding container 9d6fb2a91aacd8576ac94829dd54b18c9485f467ed31c5844e90b39f08c7c0ec: Status 404 returned error can't find the container with id 9d6fb2a91aacd8576ac94829dd54b18c9485f467ed31c5844e90b39f08c7c0ec
Sep 10 11:33:33.198380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.198354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" event={"ID":"9569d44c-5aa3-4ea0-b768-77bba04b7acb","Type":"ContainerStarted","Data":"9d6fb2a91aacd8576ac94829dd54b18c9485f467ed31c5844e90b39f08c7c0ec"}
Sep 10 11:33:33.381658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.381623    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:33:33.385236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.385217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.389349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.389319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-52dd3973-openshift-default-dockercfg-stxk6\""
Sep 10 11:33:33.397213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.397186    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:33:33.415515 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.415482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:33.521109 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8b9\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.521454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.521395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.622907 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.622869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.622914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.622935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.622960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8b9\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623421 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623886 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.623993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.623939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.625455 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.625435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.625755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.625736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.632138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.632069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.632303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.632286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8b9\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9\") pod \"e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") " pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.697389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.697353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:33.842127 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.842064    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:33.977567 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.977537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:33:33.979107 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:33:33.979067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef217bb_f42d_429d_88fc_7a80d68b81a8.slice/crio-f4760469e033d1ee881e5e0336eedddc470b847ac794c022e18a2179b945a99d WatchSource:0}: Error finding container f4760469e033d1ee881e5e0336eedddc470b847ac794c022e18a2179b945a99d: Status 404 returned error can't find the container with id f4760469e033d1ee881e5e0336eedddc470b847ac794c022e18a2179b945a99d
Sep 10 11:33:33.981619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.981577    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:33.981702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.981668    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:33.981741 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:33.981710    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:34.206651 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.206615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" event={"ID":"9569d44c-5aa3-4ea0-b768-77bba04b7acb","Type":"ContainerStarted","Data":"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"}
Sep 10 11:33:34.208615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.208589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerStarted","Data":"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"}
Sep 10 11:33:34.208749 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.208632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerStarted","Data":"f4760469e033d1ee881e5e0336eedddc470b847ac794c022e18a2179b945a99d"}
Sep 10 11:33:34.261976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.261885    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" podStartSLOduration=2.261869596 podStartE2EDuration="2.261869596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:33:34.234319421 +0000 UTC m=+1132.309645096" watchObservedRunningTime="2026-09-10 11:33:34.261869596 +0000 UTC m=+1132.337195311"
Sep 10 11:33:34.262150 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.261971    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podStartSLOduration=1.261965366 podStartE2EDuration="1.261965366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:33:34.260846835 +0000 UTC m=+1132.336172516" watchObservedRunningTime="2026-09-10 11:33:34.261965366 +0000 UTC m=+1132.337291030"
Sep 10 11:33:34.698193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:34.698151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:35.703055 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:35.703023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:36.213770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:36.213735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:36.220630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:36.220598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:36.221670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:36.221645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:36.313145 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:36.313113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:37.387592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:37.387561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:37.488653 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:37.488618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:37.586617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:37.586581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:38.062564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:38.062525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:38.163659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:38.163619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:33:42.564304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:42.564272    2578 scope.go:117] "RemoveContainer" containerID="d468078a8531746b1f674abe446880afbc45b1ee64cb8e6f2052ef113c167aec"
Sep 10 11:33:42.877046 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:42.877013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:42.877259 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:42.877068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:42.890269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:42.890233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:43.274804 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:43.274717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:33:57.353339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.353306    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerID="cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae" exitCode=0
Sep 10 11:33:57.353812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.353383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerDied","Data":"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"}
Sep 10 11:33:57.353812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.353786    2578 scope.go:117] "RemoveContainer" containerID="cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"
Sep 10 11:33:57.355035 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.355004    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:57.355127 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.355071    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:57.355172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:57.355149    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:33:58.360511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:58.360476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerStarted","Data":"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"}
Sep 10 11:33:58.698291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:58.698064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:33:58.698462 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:58.698437    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:33:58.698511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:58.698480    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:33:59.698439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:59.698391    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:33:59.698970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:33:59.698464    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:34:00.698439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:00.698395    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:34:00.698757 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:00.698723    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:34:01.241525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:01.241474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:34:01.699206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:01.698712    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:34:01.699206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:01.698793    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:34:02.698317 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:02.698278    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:34:02.698618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:02.698591    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:34:03.698450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:03.698350    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 11:34:03.698450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:03.698425    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 11:34:04.702623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:04.702593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:34:04.703068 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:04.702884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:34:04.703729 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:04.703709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:34:21.542670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.542632    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:34:21.546160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.546141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.550845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.550817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-52dd3973\"/\"e2e-lifecycle-model-52dd3973-kserve-self-signed-certs\""
Sep 10 11:34:21.550845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.550833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-52dd3973\"/\"kube-root-ca.crt\""
Sep 10 11:34:21.551056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.550964    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-52dd3973\"/\"openshift-service-ca.crt\""
Sep 10 11:34:21.551056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.550975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-52dd3973\"/\"default-dockercfg-wz2gt\""
Sep 10 11:34:21.557703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.557679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:34:21.611535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6n7f\" (UniqueName: \"kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.611738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.611738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.611738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.611900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.611900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.611821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.668696 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.668625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:34:21.713101 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713263 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713263 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6n7f\" (UniqueName: \"kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713589 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.713720 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.713666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.715619 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.715585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.715924 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.715905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.725806 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.725779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6n7f\" (UniqueName: \"kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f\") pod \"e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") " pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:21.856021 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:21.855995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:22.132795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:22.132724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:34:22.135380 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:34:22.135354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3590da55_7373_4411_81fe_01bd67ab6e4c.slice/crio-44f5c05227ea1d9d9e2b8bf519cb172f18913e6d37d7cf7b4d7ec381ff824e9b WatchSource:0}: Error finding container 44f5c05227ea1d9d9e2b8bf519cb172f18913e6d37d7cf7b4d7ec381ff824e9b: Status 404 returned error can't find the container with id 44f5c05227ea1d9d9e2b8bf519cb172f18913e6d37d7cf7b4d7ec381ff824e9b
Sep 10 11:34:22.499439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:22.499354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" event={"ID":"3590da55-7373-4411-81fe-01bd67ab6e4c","Type":"ContainerStarted","Data":"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"}
Sep 10 11:34:22.499439 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:22.499400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" event={"ID":"3590da55-7373-4411-81fe-01bd67ab6e4c","Type":"ContainerStarted","Data":"44f5c05227ea1d9d9e2b8bf519cb172f18913e6d37d7cf7b4d7ec381ff824e9b"}
Sep 10 11:34:22.522524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:22.522473    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" podStartSLOduration=1.522458013 podStartE2EDuration="1.522458013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:34:22.519335855 +0000 UTC m=+1180.594661520" watchObservedRunningTime="2026-09-10 11:34:22.522458013 +0000 UTC m=+1180.597783677"
Sep 10 11:34:24.697342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:24.697306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:34:31.857242 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:31.857208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:31.857719 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:31.857261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:31.869915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:31.869889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:32.566561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:32.566529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:34:42.468234 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:42.468199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:34:42.469606 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:42.469581    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:34:42.585976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:42.585945    2578 scope.go:117] "RemoveContainer" containerID="37b1436621149281033c020a4d6fcf51cfe0a0b0e658a4590ca85159b678827d"
Sep 10 11:34:51.189614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.189579    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"]
Sep 10 11:34:51.194432 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.194413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:51.198980 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.198960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:34:51.198980 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.198968    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:34:51.205058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.205030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"]
Sep 10 11:34:51.315630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.315593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:34:51.316795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.316776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s\" (UID: \"ce32aeae-9344-4b2e-b131-bd495b8c27eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:51.418298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.418262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s\" (UID: \"ce32aeae-9344-4b2e-b131-bd495b8c27eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:51.418867 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.418847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s\" (UID: \"ce32aeae-9344-4b2e-b131-bd495b8c27eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:51.505528 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.505446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:51.786862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:51.786834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"]
Sep 10 11:34:51.788378 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:34:51.788349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce32aeae_9344_4b2e_b131_bd495b8c27eb.slice/crio-2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7 WatchSource:0}: Error finding container 2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7: Status 404 returned error can't find the container with id 2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7
Sep 10 11:34:52.666853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:52.666813    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce32aeae-9344-4b2e-b131-bd495b8c27eb" containerID="5e71e62dbe6461cb712702e856f7ed80541fcaeb22bcf25e01145826ae5c374b" exitCode=0
Sep 10 11:34:52.667277 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:52.666901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s" event={"ID":"ce32aeae-9344-4b2e-b131-bd495b8c27eb","Type":"ContainerDied","Data":"5e71e62dbe6461cb712702e856f7ed80541fcaeb22bcf25e01145826ae5c374b"}
Sep 10 11:34:52.667277 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:52.666961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s" event={"ID":"ce32aeae-9344-4b2e-b131-bd495b8c27eb","Type":"ContainerStarted","Data":"2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7"}
Sep 10 11:34:53.814482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:53.814452    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:53.944734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:53.944647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca\") pod \"ce32aeae-9344-4b2e-b131-bd495b8c27eb\" (UID: \"ce32aeae-9344-4b2e-b131-bd495b8c27eb\") "
Sep 10 11:34:53.945041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:53.945020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce32aeae-9344-4b2e-b131-bd495b8c27eb" (UID: "ce32aeae-9344-4b2e-b131-bd495b8c27eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:34:54.045831 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:54.045785    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce32aeae-9344-4b2e-b131-bd495b8c27eb-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:34:54.680058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:54.680033    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"
Sep 10 11:34:54.680241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:54.680028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s" event={"ID":"ce32aeae-9344-4b2e-b131-bd495b8c27eb","Type":"ContainerDied","Data":"2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7"}
Sep 10 11:34:54.680241 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:34:54.680135    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6ff70d88e41833fd5e29943d63ef119f26d5e000ab59eea0d2c5829b95cee7"
Sep 10 11:35:06.871260 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:06.871217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:07.022196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:07.022158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:07.122412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:07.122296    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"]
Sep 10 11:35:07.122649 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:07.122494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:07.125368 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:07.125345    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52dd3973-g9k8s"]
Sep 10 11:35:07.202050 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:07.202017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:08.456845 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:08.456815    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce32aeae-9344-4b2e-b131-bd495b8c27eb" path="/var/lib/kubelet/pods/ce32aeae-9344-4b2e-b131-bd495b8c27eb/volumes"
Sep 10 11:35:10.497771 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:10.497731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:15.466543 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:15.466510    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:35:15.519132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:15.519099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:29.019748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:29.019715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:30.486943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.486905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" podUID="3590da55-7373-4411-81fe-01bd67ab6e4c" containerName="main" containerID="cri-o://56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa" gracePeriod=45
Sep 10 11:35:30.750887 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.750813    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:35:30.833814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.833782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.833992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.833824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.833992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.833867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.833992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.833915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.833992 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.833943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.834269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6n7f\" (UniqueName: \"kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f\") pod \"3590da55-7373-4411-81fe-01bd67ab6e4c\" (UID: \"3590da55-7373-4411-81fe-01bd67ab6e4c\") "
Sep 10 11:35:30.834269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:35:30.834269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home" (OuterVolumeSpecName: "home") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:35:30.834269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:35:30.834471 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834446    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:30.834590 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834473    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:30.834650 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.834547    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:30.836263 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.836227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm" (OuterVolumeSpecName: "dshm") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:35:30.836355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.836259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:35:30.836435 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.836415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f" (OuterVolumeSpecName: "kube-api-access-x6n7f") pod "3590da55-7373-4411-81fe-01bd67ab6e4c" (UID: "3590da55-7373-4411-81fe-01bd67ab6e4c"). InnerVolumeSpecName "kube-api-access-x6n7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:35:30.877485 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.877453    2578 generic.go:356] "Generic (PLEG): container finished" podID="3590da55-7373-4411-81fe-01bd67ab6e4c" containerID="56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa" exitCode=0
Sep 10 11:35:30.877631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.877500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"
Sep 10 11:35:30.877631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.877533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" event={"ID":"3590da55-7373-4411-81fe-01bd67ab6e4c","Type":"ContainerDied","Data":"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"}
Sep 10 11:35:30.877631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.877577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f" event={"ID":"3590da55-7373-4411-81fe-01bd67ab6e4c","Type":"ContainerDied","Data":"44f5c05227ea1d9d9e2b8bf519cb172f18913e6d37d7cf7b4d7ec381ff824e9b"}
Sep 10 11:35:30.877631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.877594    2578 scope.go:117] "RemoveContainer" containerID="56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"
Sep 10 11:35:30.890056 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.890036    2578 scope.go:117] "RemoveContainer" containerID="56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"
Sep 10 11:35:30.890387 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:35:30.890369    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa\": container with ID starting with 56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa not found: ID does not exist" containerID="56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"
Sep 10 11:35:30.890444 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.890395    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa"} err="failed to get container status \"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa\": rpc error: code = NotFound desc = could not find container \"56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa\": container with ID starting with 56e6591726dd6a1a84ef92b509430e8f7bafc30140eb1321921b9f02614689fa not found: ID does not exist"
Sep 10 11:35:30.902025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.902001    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:35:30.908174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.908140    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-52dd3973/e2e-lifecycle-model-52dd3973-kserve-69d549c5cf-ppf4f"]
Sep 10 11:35:30.935533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.935505    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3590da55-7373-4411-81fe-01bd67ab6e4c-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:30.935533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.935531    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6n7f\" (UniqueName: \"kubernetes.io/projected/3590da55-7373-4411-81fe-01bd67ab6e4c-kube-api-access-x6n7f\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:30.935693 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:30.935541    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3590da55-7373-4411-81fe-01bd67ab6e4c-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:35:32.459336 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:32.459294    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3590da55-7373-4411-81fe-01bd67ab6e4c" path="/var/lib/kubelet/pods/3590da55-7373-4411-81fe-01bd67ab6e4c/volumes"
Sep 10 11:35:42.608213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:42.608144    2578 scope.go:117] "RemoveContainer" containerID="679881f643d231bc08390068ac044deb0b1d9dd9b7c7827dbeb7ff39155af8f4"
Sep 10 11:35:42.616949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:42.616930    2578 scope.go:117] "RemoveContainer" containerID="1f93365ef6da13476d08619dca8324e31f5dffcb90ae8e5525f0cf09c8cc2ac4"
Sep 10 11:35:42.625011 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:42.624989    2578 scope.go:117] "RemoveContainer" containerID="d1a4ed43964e31b1f7dcec83104134ed4b688854208ab65c35584f4114b5b224"
Sep 10 11:35:46.581731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:46.581697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:47.542659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:47.542626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:35:47.558623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:47.558571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:53.142551 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:53.142519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:53.440204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:53.440117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:56.299622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:56.299585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:56.400156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:56.400078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:59.765994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:59.765962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:35:59.867852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:35:59.867820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:00.038476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:00.038388    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:36:00.038648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:00.038631    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" containerID="cri-o://0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83" gracePeriod=30
Sep 10 11:36:00.068994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:00.068964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:01.403985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:01.403944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:01.515966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:01.515932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:02.241758 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.241724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:02.374625 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.374586    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:02.563030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.562941    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" podUID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" containerName="main" containerID="cri-o://7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc" gracePeriod=45
Sep 10 11:36:02.825963 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.825907    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:36:02.949369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949333    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949369 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttrqc\" (UniqueName: \"kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949640    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:02.949702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache\") pod \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\" (UID: \"9569d44c-5aa3-4ea0-b768-77bba04b7acb\") "
Sep 10 11:36:02.949877 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home" (OuterVolumeSpecName: "home") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:02.950008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.949978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache" (OuterVolumeSpecName: "model-cache") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:02.950155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.950040    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:02.950155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.950058    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:02.951800 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.951780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm" (OuterVolumeSpecName: "dshm") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:02.951868 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.951838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc" (OuterVolumeSpecName: "kube-api-access-ttrqc") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "kube-api-access-ttrqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:36:02.951907 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:02.951895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9569d44c-5aa3-4ea0-b768-77bba04b7acb" (UID: "9569d44c-5aa3-4ea0-b768-77bba04b7acb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:36:03.051332 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.051298    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:03.051332 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.051326    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9569d44c-5aa3-4ea0-b768-77bba04b7acb-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:03.051332 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.051336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrqc\" (UniqueName: \"kubernetes.io/projected/9569d44c-5aa3-4ea0-b768-77bba04b7acb-kube-api-access-ttrqc\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:03.051560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.051345    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9569d44c-5aa3-4ea0-b768-77bba04b7acb-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:03.057976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.057951    2578 generic.go:356] "Generic (PLEG): container finished" podID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" containerID="7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc" exitCode=0
Sep 10 11:36:03.058143 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.057998    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"
Sep 10 11:36:03.058143 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.058044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" event={"ID":"9569d44c-5aa3-4ea0-b768-77bba04b7acb","Type":"ContainerDied","Data":"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"}
Sep 10 11:36:03.058143 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.058119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq" event={"ID":"9569d44c-5aa3-4ea0-b768-77bba04b7acb","Type":"ContainerDied","Data":"9d6fb2a91aacd8576ac94829dd54b18c9485f467ed31c5844e90b39f08c7c0ec"}
Sep 10 11:36:03.058371 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.058144    2578 scope.go:117] "RemoveContainer" containerID="7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"
Sep 10 11:36:03.067745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.067730    2578 scope.go:117] "RemoveContainer" containerID="7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"
Sep 10 11:36:03.067993 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:36:03.067974    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc\": container with ID starting with 7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc not found: ID does not exist" containerID="7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"
Sep 10 11:36:03.068039 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.068002    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc"} err="failed to get container status \"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc\": rpc error: code = NotFound desc = could not find container \"7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc\": container with ID starting with 7aeb632e25f9baa42151c5c6aae55ed21375f64c743a8d757eadb14b606aeadc not found: ID does not exist"
Sep 10 11:36:03.093390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.093331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:36:03.095281 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.095262    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/auth-test-model-13c392-kserve-57474bf699-v8qgq"]
Sep 10 11:36:03.792479 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:03.792449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:04.455728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:04.455694    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" path="/var/lib/kubelet/pods/9569d44c-5aa3-4ea0-b768-77bba04b7acb/volumes"
Sep 10 11:36:04.705688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:04.705653    2578 patch_prober.go:28] interesting pod/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:36:04.705872 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:04.705722    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:36:05.288626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.288599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:36:05.374800 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.374770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.374974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.374846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.374974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.374867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl8b9\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.374974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.374884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.374974 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.374909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.375199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.375199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:05.375199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375061    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.375199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.375402 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data\") pod \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\" (UID: \"5ef217bb-f42d-429d-88fc-7a80d68b81a8\") "
Sep 10 11:36:05.375402 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:05.375505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:36:05.375615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:05.375788 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375683    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.376031 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375798    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.376031 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375813    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.376031 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.375604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:05.377354 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.377334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:36:05.377470 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.377448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:36:05.377715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.377693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9" (OuterVolumeSpecName: "kube-api-access-wl8b9") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "kube-api-access-wl8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:36:05.377807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.377774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ef217bb-f42d-429d-88fc-7a80d68b81a8" (UID: "5ef217bb-f42d-429d-88fc-7a80d68b81a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:36:05.477055 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477024    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.477055 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477049    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.477055 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477057    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.477288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477065    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8b9\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-kube-api-access-wl8b9\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.477288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477073    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:05.477288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:05.477080    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef217bb-f42d-429d-88fc-7a80d68b81a8-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:36:06.077550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.077516    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerID="0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83" exitCode=0
Sep 10 11:36:06.077747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.077573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"
Sep 10 11:36:06.077747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.077577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerDied","Data":"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"}
Sep 10 11:36:06.077747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.077681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv" event={"ID":"5ef217bb-f42d-429d-88fc-7a80d68b81a8","Type":"ContainerDied","Data":"f4760469e033d1ee881e5e0336eedddc470b847ac794c022e18a2179b945a99d"}
Sep 10 11:36:06.077747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.077704    2578 scope.go:117] "RemoveContainer" containerID="0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"
Sep 10 11:36:06.087786 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.087766    2578 scope.go:117] "RemoveContainer" containerID="cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"
Sep 10 11:36:06.095843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.095827    2578 scope.go:117] "RemoveContainer" containerID="0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"
Sep 10 11:36:06.096142 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:36:06.096122    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83\": container with ID starting with 0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83 not found: ID does not exist" containerID="0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"
Sep 10 11:36:06.096198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.096151    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83"} err="failed to get container status \"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83\": rpc error: code = NotFound desc = could not find container \"0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83\": container with ID starting with 0f2e811967cf77aa1e3abb6ee08356e9cc887f2abb051f28ff96be2bed68bd83 not found: ID does not exist"
Sep 10 11:36:06.096198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.096169    2578 scope.go:117] "RemoveContainer" containerID="cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"
Sep 10 11:36:06.096415 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:36:06.096397    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae\": container with ID starting with cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae not found: ID does not exist" containerID="cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"
Sep 10 11:36:06.096469 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.096421    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae"} err="failed to get container status \"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae\": rpc error: code = NotFound desc = could not find container \"cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae\": container with ID starting with cd4ee453aba81e4bd6e77322f7875a54e0d348535d935c52213d6b05bef9f7ae not found: ID does not exist"
Sep 10 11:36:06.109095 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.109067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:36:06.115652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.115632    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-52dd3973-openshift-default-554fdd8584-gnzhv"]
Sep 10 11:36:06.455608 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:06.455572    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" path="/var/lib/kubelet/pods/5ef217bb-f42d-429d-88fc-7a80d68b81a8/volumes"
Sep 10 11:36:10.348183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:10.348149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:10.434430 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:10.434397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:11.031570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:11.031536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:17.585161 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:17.585126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:17.700685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:17.700654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:17.811688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:17.811652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:28.332815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.332773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:28.564302 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564266    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:36:28.564821 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564801    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:28.564870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:28.564870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564856    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" containerName="main"
Sep 10 11:36:28.564870 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564864    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" containerName="main"
Sep 10 11:36:28.564964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3590da55-7373-4411-81fe-01bd67ab6e4c" containerName="main"
Sep 10 11:36:28.564964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564906    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3590da55-7373-4411-81fe-01bd67ab6e4c" containerName="main"
Sep 10 11:36:28.564964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564915    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce32aeae-9344-4b2e-b131-bd495b8c27eb" containerName="revoke-keys"
Sep 10 11:36:28.564964 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.564924    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce32aeae-9344-4b2e-b131-bd495b8c27eb" containerName="revoke-keys"
Sep 10 11:36:28.565134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.565009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:28.565134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.565019    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce32aeae-9344-4b2e-b131-bd495b8c27eb" containerName="revoke-keys"
Sep 10 11:36:28.565134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.565030    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9569d44c-5aa3-4ea0-b768-77bba04b7acb" containerName="main"
Sep 10 11:36:28.565134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.565042    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3590da55-7373-4411-81fe-01bd67ab6e4c" containerName="main"
Sep 10 11:36:28.568547 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.568526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.569764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.569741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:28.572924 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.572904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"sub-test-model-13c392-kserve-self-signed-certs\""
Sep 10 11:36:28.573023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.572939    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"kube-root-ca.crt\""
Sep 10 11:36:28.573023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.572956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"default-dockercfg-7wps5\""
Sep 10 11:36:28.573023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.573004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13c392\"/\"openshift-service-ca.crt\""
Sep 10 11:36:28.579219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.579201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:36:28.645854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.645816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:28.700133 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.700329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f9p\" (UniqueName: \"kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.700329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.700329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.700329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.700498 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.700342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.800874 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.800874 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f9p\" (UniqueName: \"kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.800983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801392 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.801321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801392 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.801366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.801467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.801428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.803448 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.803426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.803601 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.803584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.810642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.810618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f9p\" (UniqueName: \"kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p\") pod \"sub-test-model-13c392-kserve-6777ccdfdf-qt9p2\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") " pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:28.880890 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:28.880846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:29.169615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:29.169588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:36:29.171460 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:36:29.171436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcae1849f_6980_4fd0_b468_169bd9a1170b.slice/crio-2e51ce33e1fdf814e1604a1e9a9b3aeb89873c52ca9f8aa0472ef0e6f9db5a77 WatchSource:0}: Error finding container 2e51ce33e1fdf814e1604a1e9a9b3aeb89873c52ca9f8aa0472ef0e6f9db5a77: Status 404 returned error can't find the container with id 2e51ce33e1fdf814e1604a1e9a9b3aeb89873c52ca9f8aa0472ef0e6f9db5a77
Sep 10 11:36:29.217252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:29.217222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" event={"ID":"cae1849f-6980-4fd0-b468-169bd9a1170b","Type":"ContainerStarted","Data":"2e51ce33e1fdf814e1604a1e9a9b3aeb89873c52ca9f8aa0472ef0e6f9db5a77"}
Sep 10 11:36:30.225865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:30.225828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" event={"ID":"cae1849f-6980-4fd0-b468-169bd9a1170b","Type":"ContainerStarted","Data":"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"}
Sep 10 11:36:30.249250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:30.249197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" podStartSLOduration=2.249179922 podStartE2EDuration="2.249179922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:36:30.247929323 +0000 UTC m=+1308.323254987" watchObservedRunningTime="2026-09-10 11:36:30.249179922 +0000 UTC m=+1308.324505586"
Sep 10 11:36:35.460152 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:35.460065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:36.904505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:36.904471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:38.881509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:38.881467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:38.881921 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:38.881521    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:38.894199 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:38.894171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:39.285748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:39.285649    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:36:42.656839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:42.656807    2578 scope.go:117] "RemoveContainer" containerID="365ba81ef789c7ab10021e01790824a091d2cc82965774615b389b2ab645b119"
Sep 10 11:36:42.665536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:42.665514    2578 scope.go:117] "RemoveContainer" containerID="0104c6efa4e68ab1e310bb294f06ec3f1e0530d267b8a12fcbae925568f2cfe3"
Sep 10 11:36:42.673518 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:42.673497    2578 scope.go:117] "RemoveContainer" containerID="2448edacb8af52041ffa248b0d1b333722d531944d93876aaadffd44b8015300"
Sep 10 11:36:49.525746 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:49.525711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:56.031598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.031558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:56.135008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.134972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:56.408871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.408837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:56.692623 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.692540    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:36:56.692962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.692948    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:56.693010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.692965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:56.693053 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.693024    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef217bb-f42d-429d-88fc-7a80d68b81a8" containerName="istio-proxy"
Sep 10 11:36:56.700537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.700513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.703990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.703971    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e89227-openshift-default-dockercfg-6cxf8\""
Sep 10 11:36:56.710960 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.710930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:36:56.727923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.727892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:56.762356 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762356 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqdx\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762585 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.762773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.762633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863508 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqdx\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863715 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.863998 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.863930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.864243 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.864125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.864243 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.864187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.864388 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.864365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.864467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.864447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.866303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.866281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.866415 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.866338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.873298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.873272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:56.873398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:56.873363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqdx\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx\") pod \"e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") " pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:57.011811 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.011717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:57.053209 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.053181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:57.296728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.296694    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:36:57.297235 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:36:57.297211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c4c6ef_a8ae_4e56_932f_18ce66c3a2bf.slice/crio-78c4aadae8635a1a985b3d4d30006674c4a9a801e8ecfa0d82b07c54f3a8c9b8 WatchSource:0}: Error finding container 78c4aadae8635a1a985b3d4d30006674c4a9a801e8ecfa0d82b07c54f3a8c9b8: Status 404 returned error can't find the container with id 78c4aadae8635a1a985b3d4d30006674c4a9a801e8ecfa0d82b07c54f3a8c9b8
Sep 10 11:36:57.299875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.299834    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:36:57.299968 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.299923    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:36:57.300009 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.299969    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:36:57.377637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.377589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerStarted","Data":"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"}
Sep 10 11:36:57.377834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.377697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerStarted","Data":"78c4aadae8635a1a985b3d4d30006674c4a9a801e8ecfa0d82b07c54f3a8c9b8"}
Sep 10 11:36:57.404614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:57.404551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podStartSLOduration=1.404533057 podStartE2EDuration="1.404533057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:36:57.403048965 +0000 UTC m=+1335.478374630" watchObservedRunningTime="2026-09-10 11:36:57.404533057 +0000 UTC m=+1335.479858721"
Sep 10 11:36:58.012572 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:58.012527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:58.013990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:58.013958    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:36:58.014160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:58.014019    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:36:59.017146 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:59.017080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:59.389063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:59.389029    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:59.390552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:59.390531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:36:59.399334 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:59.399310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:36:59.513284 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:36:59.513249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:00.712925 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:00.712889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:00.797516 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:00.797480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:01.253190 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.253150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:01.352679 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.352648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:01.442888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.442838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:01.483122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.483064    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:37:01.486817 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.486792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.490702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.490666    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e89227\""
Sep 10 11:37:01.491030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.491011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e89227-dockercfg-7b78p\""
Sep 10 11:37:01.500833 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.500806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:37:01.614795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.614765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.614970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.614899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8np\" (UniqueName: \"kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.716052 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.716009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8np\" (UniqueName: \"kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.716473 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.716125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.716878 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.716859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.726094 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.726054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8np\" (UniqueName: \"kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np\") pod \"payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:01.799769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:01.799734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:02.075227 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:02.075200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:37:02.076732 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:37:02.076708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3769501_daed_4f89_9691_0f3ac1b0afb9.slice/crio-7cf7a2c17b56269794841711e0601fc934d7204a281c395b82aa4cdfdd85ecd7 WatchSource:0}: Error finding container 7cf7a2c17b56269794841711e0601fc934d7204a281c395b82aa4cdfdd85ecd7: Status 404 returned error can't find the container with id 7cf7a2c17b56269794841711e0601fc934d7204a281c395b82aa4cdfdd85ecd7
Sep 10 11:37:02.405620 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:02.405577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" event={"ID":"c3769501-daed-4f89-9691-0f3ac1b0afb9","Type":"ContainerStarted","Data":"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"}
Sep 10 11:37:02.405929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:02.405631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" event={"ID":"c3769501-daed-4f89-9691-0f3ac1b0afb9","Type":"ContainerStarted","Data":"7cf7a2c17b56269794841711e0601fc934d7204a281c395b82aa4cdfdd85ecd7"}
Sep 10 11:37:02.405929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:02.405666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:02.428581 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:02.427947    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" podStartSLOduration=1.427925205 podStartE2EDuration="1.427925205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:37:02.426286506 +0000 UTC m=+1340.501612172" watchObservedRunningTime="2026-09-10 11:37:02.427925205 +0000 UTC m=+1340.503250886"
Sep 10 11:37:13.414029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:13.413949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:37:17.497378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.497345    2578 generic.go:356] "Generic (PLEG): container finished" podID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerID="f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92" exitCode=0
Sep 10 11:37:17.497784 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.497426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerDied","Data":"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"}
Sep 10 11:37:17.497834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.497820    2578 scope.go:117] "RemoveContainer" containerID="f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"
Sep 10 11:37:17.498926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.498899    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:17.498990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.498964    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:17.499026 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:17.498995    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:18.505613 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:18.505584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerStarted","Data":"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"}
Sep 10 11:37:19.012464 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:19.012422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:37:19.012820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:19.012791    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:19.012895 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:19.012870    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:20.012680 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:20.012642    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:20.013061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:20.012698    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:21.012392 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:21.012348    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:21.012657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:21.012412    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:22.012762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:22.012714    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:22.013277 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:22.012787    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:23.013045 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:23.013002    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:23.013546 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:23.013062    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:23.845621 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:23.845593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:24.012731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.012691    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 11:37:24.013045 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.012777    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 11:37:24.371080 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.371045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:24.872969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.872938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:24.960731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.960697    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:37:24.987638 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:24.987602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:25.016574 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.016546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:37:25.016934 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.016919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:37:25.017554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.017540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:37:25.205170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.205073    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:37:25.208672 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.208655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.212993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.212973    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7a970e-openshift-default-dockercfg-xxkgq\""
Sep 10 11:37:25.223490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.223467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:37:25.245864 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.245827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:25.362066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdp4c\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.362446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.362441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.463554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.463554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdp4c\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.463972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464478 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.464234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464478 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.464268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.464549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.464531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.466072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.466043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.466298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.466278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.478024 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.478000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.479929 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.479903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdp4c\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c\") pod \"e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") " pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.481254 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.481233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:25.519474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.519440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:25.805322 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.805148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:37:25.808138 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:37:25.808106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b79e358_0209_4652_bd19_304a3fdeb0f4.slice/crio-88bbe51e4fea8e523ef916e4552ca03e545ea9caf27c32c13438800b40c490bf WatchSource:0}: Error finding container 88bbe51e4fea8e523ef916e4552ca03e545ea9caf27c32c13438800b40c490bf: Status 404 returned error can't find the container with id 88bbe51e4fea8e523ef916e4552ca03e545ea9caf27c32c13438800b40c490bf
Sep 10 11:37:25.810293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.810259    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:25.810386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.810324    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:25.810386 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:25.810357    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:26.555859 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:26.555821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerStarted","Data":"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"}
Sep 10 11:37:26.556398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:26.555875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerStarted","Data":"88bbe51e4fea8e523ef916e4552ca03e545ea9caf27c32c13438800b40c490bf"}
Sep 10 11:37:26.584361 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:26.584313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podStartSLOduration=1.584298258 podStartE2EDuration="1.584298258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:37:26.581048112 +0000 UTC m=+1364.656373774" watchObservedRunningTime="2026-09-10 11:37:26.584298258 +0000 UTC m=+1364.659623921"
Sep 10 11:37:27.520023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:27.519987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:27.525183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:27.525157    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:27.561341 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:27.561314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:27.562218 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:27.562198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:27.988991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:27.988949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:28.588856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:28.588819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:28.683824 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:28.683778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:29.779642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:29.779593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:29.878358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:29.878323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:29.987364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:29.987327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:30.411634 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.411603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:30.506686 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.506652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:30.622731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.622682    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:37:30.626514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.626496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.630246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.630222    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7a970e-dockercfg-bb74q\""
Sep 10 11:37:30.630246 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.630240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7a970e\""
Sep 10 11:37:30.645074 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.645046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:37:30.715171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.715052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.715487 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.715462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqll\" (UniqueName: \"kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.816711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.816663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.817162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.816729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqll\" (UniqueName: \"kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.817419 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.817398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.826078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.826060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqll\" (UniqueName: \"kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll\") pod \"payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:30.937593 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:30.937562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:31.214711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:31.214688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:37:31.216338 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:37:31.216296    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36cf3be_e95e_42ea_bb01_c6ee1251d009.slice/crio-0a28fde26aa5680aa403dbc9665c5ed0d85d2ea4b1daf3f0b473d8797cf5e782 WatchSource:0}: Error finding container 0a28fde26aa5680aa403dbc9665c5ed0d85d2ea4b1daf3f0b473d8797cf5e782: Status 404 returned error can't find the container with id 0a28fde26aa5680aa403dbc9665c5ed0d85d2ea4b1daf3f0b473d8797cf5e782
Sep 10 11:37:31.586016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:31.585912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" event={"ID":"d36cf3be-e95e-42ea-bb01-c6ee1251d009","Type":"ContainerStarted","Data":"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"}
Sep 10 11:37:31.586016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:31.585986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" event={"ID":"d36cf3be-e95e-42ea-bb01-c6ee1251d009","Type":"ContainerStarted","Data":"0a28fde26aa5680aa403dbc9665c5ed0d85d2ea4b1daf3f0b473d8797cf5e782"}
Sep 10 11:37:31.586252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:31.586125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:31.608843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:31.608781    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" podStartSLOduration=1.608762899 podStartE2EDuration="1.608762899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:37:31.607510863 +0000 UTC m=+1369.682836526" watchObservedRunningTime="2026-09-10 11:37:31.608762899 +0000 UTC m=+1369.684088563"
Sep 10 11:37:39.980841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:39.980791    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" podUID="cae1849f-6980-4fd0-b468-169bd9a1170b" containerName="main" containerID="cri-o://9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e" gracePeriod=45
Sep 10 11:37:40.281437 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.281412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:37:40.319712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.319910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319725    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.319910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.319910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6f9p\" (UniqueName: \"kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.319910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.319910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.319908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home\") pod \"cae1849f-6980-4fd0-b468-169bd9a1170b\" (UID: \"cae1849f-6980-4fd0-b468-169bd9a1170b\") "
Sep 10 11:37:40.320236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:37:40.320236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:37:40.320236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home" (OuterVolumeSpecName: "home") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:37:40.320393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320326    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.320393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320341    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.320393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.320376    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.322446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.322415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm" (OuterVolumeSpecName: "dshm") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:37:40.322630 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.322604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p" (OuterVolumeSpecName: "kube-api-access-g6f9p") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "kube-api-access-g6f9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:37:40.323231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.323200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cae1849f-6980-4fd0-b468-169bd9a1170b" (UID: "cae1849f-6980-4fd0-b468-169bd9a1170b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:37:40.421770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.421734    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f9p\" (UniqueName: \"kubernetes.io/projected/cae1849f-6980-4fd0-b468-169bd9a1170b-kube-api-access-g6f9p\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.421770 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.421770    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae1849f-6980-4fd0-b468-169bd9a1170b-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.421988 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.421784    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae1849f-6980-4fd0-b468-169bd9a1170b-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:37:40.638073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.638038    2578 generic.go:356] "Generic (PLEG): container finished" podID="cae1849f-6980-4fd0-b468-169bd9a1170b" containerID="9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e" exitCode=0
Sep 10 11:37:40.638073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.638080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" event={"ID":"cae1849f-6980-4fd0-b468-169bd9a1170b","Type":"ContainerDied","Data":"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"}
Sep 10 11:37:40.638366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.638121    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"
Sep 10 11:37:40.638366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.638139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2" event={"ID":"cae1849f-6980-4fd0-b468-169bd9a1170b","Type":"ContainerDied","Data":"2e51ce33e1fdf814e1604a1e9a9b3aeb89873c52ca9f8aa0472ef0e6f9db5a77"}
Sep 10 11:37:40.638366 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.638156    2578 scope.go:117] "RemoveContainer" containerID="9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"
Sep 10 11:37:40.646883 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.646846    2578 scope.go:117] "RemoveContainer" containerID="9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"
Sep 10 11:37:40.647190 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:37:40.647135    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e\": container with ID starting with 9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e not found: ID does not exist" containerID="9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"
Sep 10 11:37:40.647190 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.647161    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e"} err="failed to get container status \"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e\": rpc error: code = NotFound desc = could not find container \"9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e\": container with ID starting with 9ec519cc3328daace3361f8e936113b2826e0f1b2ad2fcb95967d32120c8819e not found: ID does not exist"
Sep 10 11:37:40.661195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.661163    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:37:40.666186 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:40.666166    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13c392/sub-test-model-13c392-kserve-6777ccdfdf-qt9p2"]
Sep 10 11:37:42.457941 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:42.457909    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae1849f-6980-4fd0-b468-169bd9a1170b" path="/var/lib/kubelet/pods/cae1849f-6980-4fd0-b468-169bd9a1170b/volumes"
Sep 10 11:37:42.594463 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:42.594430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:37:43.603294 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:43.603253    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 11:37:43.603697 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:43.603340    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 11:37:44.663676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.663592    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerID="484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95" exitCode=0
Sep 10 11:37:44.664079 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.663679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerDied","Data":"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"}
Sep 10 11:37:44.664165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.664150    2578 scope.go:117] "RemoveContainer" containerID="484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"
Sep 10 11:37:44.665426 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.665395    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:44.665503 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.665468    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:44.665544 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:44.665501    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:37:45.672006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:45.671974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerStarted","Data":"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"}
Sep 10 11:37:46.519835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:46.519793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:46.520111 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:46.520060    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 11:37:46.520219 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:46.520151    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 11:37:47.520198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:47.520156    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 11:37:47.520605 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:47.520215    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 11:37:47.834999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:47.834909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:48.520664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:48.520622    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 11:37:48.521066 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:48.520688    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 11:37:49.520398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:49.520354    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 11:37:49.520618 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:49.520419    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 11:37:50.520560 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:50.520515    2578 patch_prober.go:28] interesting pod/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 11:37:50.520950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:50.520579    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 11:37:50.917179 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:50.917148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:51.517141 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:51.517106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:51.524061 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:51.524038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:51.524430 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:51.524317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:51.524923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:51.524907    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:37:53.607435 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:53.607397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:55.793981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:55.793946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:56.662535 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.662501    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"]
Sep 10 11:37:56.662919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.662906    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae1849f-6980-4fd0-b468-169bd9a1170b" containerName="main"
Sep 10 11:37:56.662969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.662921    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae1849f-6980-4fd0-b468-169bd9a1170b" containerName="main"
Sep 10 11:37:56.663004 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.662994    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae1849f-6980-4fd0-b468-169bd9a1170b" containerName="main"
Sep 10 11:37:56.666304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.666288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:37:56.669375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.669346    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:37:56.669375 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.669370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:37:56.676836 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.676809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"]
Sep 10 11:37:56.779316 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.779279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:56.797700 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.797663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e89227-hh7pf\" (UID: \"511f46e9-e3a2-44dd-ab14-a713ae2993a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:37:56.898916 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.898883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e89227-hh7pf\" (UID: \"511f46e9-e3a2-44dd-ab14-a713ae2993a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:37:56.899590 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.899569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e89227-hh7pf\" (UID: \"511f46e9-e3a2-44dd-ab14-a713ae2993a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:37:56.977966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:56.977877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:37:57.254420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:57.254394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"]
Sep 10 11:37:57.256593 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:37:57.256561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511f46e9_e3a2_44dd_ab14_a713ae2993a9.slice/crio-39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8 WatchSource:0}: Error finding container 39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8: Status 404 returned error can't find the container with id 39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8
Sep 10 11:37:57.621775 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:57.621713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:57.722370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:57.722340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:37:57.770668 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:57.770630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf" event={"ID":"511f46e9-e3a2-44dd-ab14-a713ae2993a9","Type":"ContainerStarted","Data":"39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8"}
Sep 10 11:37:58.777935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:58.777902    2578 generic.go:356] "Generic (PLEG): container finished" podID="511f46e9-e3a2-44dd-ab14-a713ae2993a9" containerID="187633b875876256952c8310aa427478b94471fd2b9bdb5070320a54c0c5475c" exitCode=0
Sep 10 11:37:58.778372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:58.777988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf" event={"ID":"511f46e9-e3a2-44dd-ab14-a713ae2993a9","Type":"ContainerDied","Data":"187633b875876256952c8310aa427478b94471fd2b9bdb5070320a54c0c5475c"}
Sep 10 11:37:59.917348 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:37:59.917319    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:38:00.033893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.033799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca\") pod \"511f46e9-e3a2-44dd-ab14-a713ae2993a9\" (UID: \"511f46e9-e3a2-44dd-ab14-a713ae2993a9\") "
Sep 10 11:38:00.034207 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.034182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "511f46e9-e3a2-44dd-ab14-a713ae2993a9" (UID: "511f46e9-e3a2-44dd-ab14-a713ae2993a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:00.135154 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.135075    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511f46e9-e3a2-44dd-ab14-a713ae2993a9-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:00.796020 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.795927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"
Sep 10 11:38:00.796020 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.795961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf" event={"ID":"511f46e9-e3a2-44dd-ab14-a713ae2993a9","Type":"ContainerDied","Data":"39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8"}
Sep 10 11:38:00.796319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:00.796052    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39463aaba559ef92c26a9cf6d8a91133f26d08779edaf153515f1bc2e6e01ae8"
Sep 10 11:38:07.451514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:07.451472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:12.299691 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.299661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:12.343401 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.343366    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:38:12.343659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.343633    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" podUID="c3769501-daed-4f89-9691-0f3ac1b0afb9" containerName="payload-pre-processing" containerID="cri-o://e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265" gracePeriod=30
Sep 10 11:38:12.399054 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.399022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:12.497765 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.497733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:12.546642 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.546601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"]
Sep 10 11:38:12.551224 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.551145    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e89227-hh7pf"]
Sep 10 11:38:12.597909 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.597884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:12.613871 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.613844    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:38:12.762411 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.762372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume\") pod \"c3769501-daed-4f89-9691-0f3ac1b0afb9\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") "
Sep 10 11:38:12.762575 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.762426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq8np\" (UniqueName: \"kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np\") pod \"c3769501-daed-4f89-9691-0f3ac1b0afb9\" (UID: \"c3769501-daed-4f89-9691-0f3ac1b0afb9\") "
Sep 10 11:38:12.762762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.762738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3769501-daed-4f89-9691-0f3ac1b0afb9" (UID: "c3769501-daed-4f89-9691-0f3ac1b0afb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:12.764681 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.764662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np" (OuterVolumeSpecName: "kube-api-access-qq8np") pod "c3769501-daed-4f89-9691-0f3ac1b0afb9" (UID: "c3769501-daed-4f89-9691-0f3ac1b0afb9"). InnerVolumeSpecName "kube-api-access-qq8np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:12.863062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863031    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3769501-daed-4f89-9691-0f3ac1b0afb9" containerID="e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265" exitCode=0
Sep 10 11:38:12.863273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"
Sep 10 11:38:12.863273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" event={"ID":"c3769501-daed-4f89-9691-0f3ac1b0afb9","Type":"ContainerDied","Data":"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"}
Sep 10 11:38:12.863273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg" event={"ID":"c3769501-daed-4f89-9691-0f3ac1b0afb9","Type":"ContainerDied","Data":"7cf7a2c17b56269794841711e0601fc934d7204a281c395b82aa4cdfdd85ecd7"}
Sep 10 11:38:12.863273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863198    2578 scope.go:117] "RemoveContainer" containerID="e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"
Sep 10 11:38:12.863273 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863258    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8np\" (UniqueName: \"kubernetes.io/projected/c3769501-daed-4f89-9691-0f3ac1b0afb9-kube-api-access-qq8np\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:12.863494 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.863377    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3769501-daed-4f89-9691-0f3ac1b0afb9-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:12.872818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.872797    2578 scope.go:117] "RemoveContainer" containerID="e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"
Sep 10 11:38:12.873129 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:12.873078    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265\": container with ID starting with e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265 not found: ID does not exist" containerID="e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"
Sep 10 11:38:12.873235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.873125    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265"} err="failed to get container status \"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265\": rpc error: code = NotFound desc = could not find container \"e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265\": container with ID starting with e61ef20e308cf6f47917ea2bc19b91670ba98209710ee4d763d5ad8cd1570265 not found: ID does not exist"
Sep 10 11:38:12.889882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.889851    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:38:12.893079 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.893053    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e89227-86d867946-mxbpg"]
Sep 10 11:38:12.928191 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:12.928159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:14.456728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:14.456689    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511f46e9-e3a2-44dd-ab14-a713ae2993a9" path="/var/lib/kubelet/pods/511f46e9-e3a2-44dd-ab14-a713ae2993a9/volumes"
Sep 10 11:38:14.457132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:14.457058    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3769501-daed-4f89-9691-0f3ac1b0afb9" path="/var/lib/kubelet/pods/c3769501-daed-4f89-9691-0f3ac1b0afb9/volumes"
Sep 10 11:38:17.590235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:17.590201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:17.979908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:17.979881    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:19.838497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:19.838456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:19.925708 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:19.925677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:20.126019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.125986    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:38:20.126267 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.126244    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" containerID="cri-o://019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad" gracePeriod=30
Sep 10 11:38:20.152958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.152920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:20.493714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.493625    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"]
Sep 10 11:38:20.494071 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511f46e9-e3a2-44dd-ab14-a713ae2993a9" containerName="revoke-keys"
Sep 10 11:38:20.494169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494074    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="511f46e9-e3a2-44dd-ab14-a713ae2993a9" containerName="revoke-keys"
Sep 10 11:38:20.494169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494100    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3769501-daed-4f89-9691-0f3ac1b0afb9" containerName="payload-pre-processing"
Sep 10 11:38:20.494169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494106    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3769501-daed-4f89-9691-0f3ac1b0afb9" containerName="payload-pre-processing"
Sep 10 11:38:20.494266 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494174    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="511f46e9-e3a2-44dd-ab14-a713ae2993a9" containerName="revoke-keys"
Sep 10 11:38:20.494266 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.494185    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3769501-daed-4f89-9691-0f3ac1b0afb9" containerName="payload-pre-processing"
Sep 10 11:38:20.498777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.498757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:20.502446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.502418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:38:20.502598 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.502542    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:38:20.511080 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.511048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"]
Sep 10 11:38:20.541060 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.541019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx\" (UID: \"19030f10-b4a0-4fe4-ad53-fb9fe20633d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:20.641768 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.641727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx\" (UID: \"19030f10-b4a0-4fe4-ad53-fb9fe20633d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:20.642393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.642368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx\" (UID: \"19030f10-b4a0-4fe4-ad53-fb9fe20633d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:20.770639 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.770530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:20.810115 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:20.810055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:21.090986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:21.090957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"]
Sep 10 11:38:21.092526 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:38:21.092493    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19030f10_b4a0_4fe4_ad53_fb9fe20633d5.slice/crio-58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d WatchSource:0}: Error finding container 58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d: Status 404 returned error can't find the container with id 58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d
Sep 10 11:38:21.917123 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:21.917073    2578 generic.go:356] "Generic (PLEG): container finished" podID="19030f10-b4a0-4fe4-ad53-fb9fe20633d5" containerID="a30e91f1c479de5cd291adcbd8cafcd219b2174faa4e9d6efacb0eea6f0adf34" exitCode=0
Sep 10 11:38:21.917287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:21.917120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx" event={"ID":"19030f10-b4a0-4fe4-ad53-fb9fe20633d5","Type":"ContainerDied","Data":"a30e91f1c479de5cd291adcbd8cafcd219b2174faa4e9d6efacb0eea6f0adf34"}
Sep 10 11:38:21.917287 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:21.917173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx" event={"ID":"19030f10-b4a0-4fe4-ad53-fb9fe20633d5","Type":"ContainerStarted","Data":"58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d"}
Sep 10 11:38:23.059920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.059891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:23.162962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.162918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca\") pod \"19030f10-b4a0-4fe4-ad53-fb9fe20633d5\" (UID: \"19030f10-b4a0-4fe4-ad53-fb9fe20633d5\") "
Sep 10 11:38:23.163383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.163360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19030f10-b4a0-4fe4-ad53-fb9fe20633d5" (UID: "19030f10-b4a0-4fe4-ad53-fb9fe20633d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:23.264713 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.264613    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19030f10-b4a0-4fe4-ad53-fb9fe20633d5-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:23.929497 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.929460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx" event={"ID":"19030f10-b4a0-4fe4-ad53-fb9fe20633d5","Type":"ContainerDied","Data":"58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d"}
Sep 10 11:38:23.929769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.929522    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a5c5902332247a4b64b8dc533ded70eb47180bbb1eb8691156ddb929170f7d"
Sep 10 11:38:23.929769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:23.929477    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"
Sep 10 11:38:24.654555 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:24.654520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:25.020165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.020061    2578 patch_prober.go:28] interesting pod/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:38:25.020165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.020155    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:38:25.388762 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.388730    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:38:25.483714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcqdx\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.483920 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.483988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs\") pod \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\" (UID: \"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf\") "
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:25.484288 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484273    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:25.484594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484469    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.484594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484499    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.484594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484514    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.484594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484527    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.484969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.484935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:25.486658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.486631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:25.486790 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.486763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:25.487009 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.486982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx" (OuterVolumeSpecName: "kube-api-access-dcqdx") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "kube-api-access-dcqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:25.487009 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.486995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" (UID: "90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:38:25.585814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.585717    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.585814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.585759    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.585814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.585773    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqdx\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-kube-api-access-dcqdx\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.585814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.585789    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.585814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.585802    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:25.942403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.942364    2578 generic.go:356] "Generic (PLEG): container finished" podID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerID="019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad" exitCode=0
Sep 10 11:38:25.942852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.942421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"
Sep 10 11:38:25.942852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.942444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerDied","Data":"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"}
Sep 10 11:38:25.942852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.942496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh" event={"ID":"90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf","Type":"ContainerDied","Data":"78c4aadae8635a1a985b3d4d30006674c4a9a801e8ecfa0d82b07c54f3a8c9b8"}
Sep 10 11:38:25.942852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.942510    2578 scope.go:117] "RemoveContainer" containerID="019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"
Sep 10 11:38:25.954606 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.954588    2578 scope.go:117] "RemoveContainer" containerID="f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"
Sep 10 11:38:25.963588 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.963568    2578 scope.go:117] "RemoveContainer" containerID="019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"
Sep 10 11:38:25.963852 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:25.963831    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad\": container with ID starting with 019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad not found: ID does not exist" containerID="019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"
Sep 10 11:38:25.963899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.963861    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad"} err="failed to get container status \"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad\": rpc error: code = NotFound desc = could not find container \"019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad\": container with ID starting with 019f9b6e62d22af081e3b4e6ee6757709d9e472078435fc0aef95f971736ccad not found: ID does not exist"
Sep 10 11:38:25.963899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.963878    2578 scope.go:117] "RemoveContainer" containerID="f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"
Sep 10 11:38:25.964172 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:25.964154    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92\": container with ID starting with f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92 not found: ID does not exist" containerID="f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"
Sep 10 11:38:25.964216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.964179    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92"} err="failed to get container status \"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92\": rpc error: code = NotFound desc = could not find container \"f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92\": container with ID starting with f19a414d86d7de02d5f7821550e9f7b826346be149c7725fe1b83a79e4c7ad92 not found: ID does not exist"
Sep 10 11:38:25.974048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.974020    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:38:25.977406 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:25.977380    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e89227-openshift-default-5fd454f558-bc2mh"]
Sep 10 11:38:26.457501 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:26.457472    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" path="/var/lib/kubelet/pods/90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf/volumes"
Sep 10 11:38:36.160880 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.160799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:36.197550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.197520    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:38:36.197771 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.197748    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" podUID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" containerName="payload-pre-processing" containerID="cri-o://6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f" gracePeriod=30
Sep 10 11:38:36.263518 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.263487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:36.387534 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.387299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:36.435300 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.435156    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"]
Sep 10 11:38:36.437062 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.437024    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a970e-rnjnx"]
Sep 10 11:38:36.456332 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.456300    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19030f10-b4a0-4fe4-ad53-fb9fe20633d5" path="/var/lib/kubelet/pods/19030f10-b4a0-4fe4-ad53-fb9fe20633d5/volumes"
Sep 10 11:38:36.479613 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.479591    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:38:36.484523 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.484496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:36.608289 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.608259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume\") pod \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") "
Sep 10 11:38:36.608461 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.608377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sqll\" (UniqueName: \"kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll\") pod \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\" (UID: \"d36cf3be-e95e-42ea-bb01-c6ee1251d009\") "
Sep 10 11:38:36.608721 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.608695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d36cf3be-e95e-42ea-bb01-c6ee1251d009" (UID: "d36cf3be-e95e-42ea-bb01-c6ee1251d009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:36.610612 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.610586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll" (OuterVolumeSpecName: "kube-api-access-9sqll") pod "d36cf3be-e95e-42ea-bb01-c6ee1251d009" (UID: "d36cf3be-e95e-42ea-bb01-c6ee1251d009"). InnerVolumeSpecName "kube-api-access-9sqll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:36.709587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.709490    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36cf3be-e95e-42ea-bb01-c6ee1251d009-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:36.709587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.709528    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqll\" (UniqueName: \"kubernetes.io/projected/d36cf3be-e95e-42ea-bb01-c6ee1251d009-kube-api-access-9sqll\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:36.820406 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:36.820371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:37.009834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.009739    2578 generic.go:356] "Generic (PLEG): container finished" podID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" containerID="6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f" exitCode=0
Sep 10 11:38:37.009834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.009788    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"
Sep 10 11:38:37.010018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.009829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" event={"ID":"d36cf3be-e95e-42ea-bb01-c6ee1251d009","Type":"ContainerDied","Data":"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"}
Sep 10 11:38:37.010018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.009881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9" event={"ID":"d36cf3be-e95e-42ea-bb01-c6ee1251d009","Type":"ContainerDied","Data":"0a28fde26aa5680aa403dbc9665c5ed0d85d2ea4b1daf3f0b473d8797cf5e782"}
Sep 10 11:38:37.010018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.009904    2578 scope.go:117] "RemoveContainer" containerID="6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"
Sep 10 11:38:37.019479 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.019460    2578 scope.go:117] "RemoveContainer" containerID="6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"
Sep 10 11:38:37.019806 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:37.019787    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f\": container with ID starting with 6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f not found: ID does not exist" containerID="6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"
Sep 10 11:38:37.019861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.019819    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f"} err="failed to get container status \"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f\": rpc error: code = NotFound desc = could not find container \"6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f\": container with ID starting with 6e3128fc62f5d012f5ba6efe89c30e7d58ffe3bf89e6f0df054c3c82349ae81f not found: ID does not exist"
Sep 10 11:38:37.032728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.032698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:38:37.037291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:37.037263    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a970e-d9f8cbbb-v2db9"]
Sep 10 11:38:38.456242 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:38.456213    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" path="/var/lib/kubelet/pods/d36cf3be-e95e-42ea-bb01-c6ee1251d009/volumes"
Sep 10 11:38:42.720466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:42.720425    2578 scope.go:117] "RemoveContainer" containerID="ea480344a91532d6f300369d1028d0184f6a138f8925be132cf4924df9985b4c"
Sep 10 11:38:43.152274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.152239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:43.227382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.227353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:43.421917 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.421822    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:38:43.422151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.422037    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy" containerID="cri-o://a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903" gracePeriod=30
Sep 10 11:38:43.453475 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.453446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:43.805311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805214    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"]
Sep 10 11:38:43.805764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805747    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:38:43.805841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805765    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:38:43.805841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" containerName="payload-pre-processing"
Sep 10 11:38:43.805841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805801    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" containerName="payload-pre-processing"
Sep 10 11:38:43.805841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805831    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19030f10-b4a0-4fe4-ad53-fb9fe20633d5" containerName="revoke-keys"
Sep 10 11:38:43.805841 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805839    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19030f10-b4a0-4fe4-ad53-fb9fe20633d5" containerName="revoke-keys"
Sep 10 11:38:43.806139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805925    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19030f10-b4a0-4fe4-ad53-fb9fe20633d5" containerName="revoke-keys"
Sep 10 11:38:43.806139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805937    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:38:43.806139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805946    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36cf3be-e95e-42ea-bb01-c6ee1251d009" containerName="payload-pre-processing"
Sep 10 11:38:43.806139 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.805960    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:38:43.810561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.810538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:43.813816 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.813792    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:38:43.813945 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.813794    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:38:43.828509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.828470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"]
Sep 10 11:38:43.920008 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.919965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:43.982616 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:43.982579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb\" (UID: \"b1dbf712-b246-4244-b3a5-093ea417de12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:44.083532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:44.083434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb\" (UID: \"b1dbf712-b246-4244-b3a5-093ea417de12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:44.084160 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:44.084141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb\" (UID: \"b1dbf712-b246-4244-b3a5-093ea417de12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:44.131936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:44.131907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:44.424795 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:44.424765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"]
Sep 10 11:38:44.425567 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:38:44.425538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dbf712_b246_4244_b3a5_093ea417de12.slice/crio-73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c WatchSource:0}: Error finding container 73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c: Status 404 returned error can't find the container with id 73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c
Sep 10 11:38:45.058100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:45.057976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb" event={"ID":"b1dbf712-b246-4244-b3a5-093ea417de12","Type":"ContainerStarted","Data":"8b4a35d43ff4ecf5ce899803b3f5b81ab2a4276bf3cd067924f2bfd1f0d37c77"}
Sep 10 11:38:45.058100 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:45.058040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb" event={"ID":"b1dbf712-b246-4244-b3a5-093ea417de12","Type":"ContainerStarted","Data":"73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c"}
Sep 10 11:38:45.080323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:45.080270    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb" podStartSLOduration=2.080254631 podStartE2EDuration="2.080254631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:38:45.078683509 +0000 UTC m=+1443.154009173" watchObservedRunningTime="2026-09-10 11:38:45.080254631 +0000 UTC m=+1443.155580298"
Sep 10 11:38:46.065622 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:46.065582    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1dbf712-b246-4244-b3a5-093ea417de12" containerID="8b4a35d43ff4ecf5ce899803b3f5b81ab2a4276bf3cd067924f2bfd1f0d37c77" exitCode=0
Sep 10 11:38:46.066016 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:46.065666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb" event={"ID":"b1dbf712-b246-4244-b3a5-093ea417de12","Type":"ContainerDied","Data":"8b4a35d43ff4ecf5ce899803b3f5b81ab2a4276bf3cd067924f2bfd1f0d37c77"}
Sep 10 11:38:47.206001 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:47.205974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:47.313504 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:47.313469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca\") pod \"b1dbf712-b246-4244-b3a5-093ea417de12\" (UID: \"b1dbf712-b246-4244-b3a5-093ea417de12\") "
Sep 10 11:38:47.313848 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:47.313824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1dbf712-b246-4244-b3a5-093ea417de12" (UID: "b1dbf712-b246-4244-b3a5-093ea417de12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:47.414766 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:47.414733    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dbf712-b246-4244-b3a5-093ea417de12-maas-api-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.080321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.080289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb" event={"ID":"b1dbf712-b246-4244-b3a5-093ea417de12","Type":"ContainerDied","Data":"73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c"}
Sep 10 11:38:48.080478 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.080340    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73203f1740a56beb82266755003b01b675249a27b91b8194dbf9487fb2e8456c"
Sep 10 11:38:48.080478 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.080304    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"
Sep 10 11:38:48.682611 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.682581    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:38:48.829936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.829852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.829936 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.829908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.829942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdp4c\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.829971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:48.830398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert\") pod \"4b79e358-0209-4652-bd19-304a3fdeb0f4\" (UID: \"4b79e358-0209-4652-bd19-304a3fdeb0f4\") "
Sep 10 11:38:48.830398 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:48.830647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:48.830647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:48.830739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830663    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.830739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830683    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.830739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830697    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.830739 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830711    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.830915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.830743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:38:48.832474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.832449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c" (OuterVolumeSpecName: "kube-api-access-vdp4c") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "kube-api-access-vdp4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:48.832474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.832457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:38:48.832893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.832867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:38:48.832893 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.832879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b79e358-0209-4652-bd19-304a3fdeb0f4" (UID: "4b79e358-0209-4652-bd19-304a3fdeb0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:38:48.931561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.931510    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdp4c\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-kube-api-access-vdp4c\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.931561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.931557    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.931561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.931568    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.931561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.931577    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b79e358-0209-4652-bd19-304a3fdeb0f4-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:48.931846 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:48.931584    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b79e358-0209-4652-bd19-304a3fdeb0f4-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:38:49.087809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.087722    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerID="a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903" exitCode=0
Sep 10 11:38:49.087809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.087772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"
Sep 10 11:38:49.087986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.087804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerDied","Data":"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"}
Sep 10 11:38:49.087986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.087859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6" event={"ID":"4b79e358-0209-4652-bd19-304a3fdeb0f4","Type":"ContainerDied","Data":"88bbe51e4fea8e523ef916e4552ca03e545ea9caf27c32c13438800b40c490bf"}
Sep 10 11:38:49.087986 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.087883    2578 scope.go:117] "RemoveContainer" containerID="a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"
Sep 10 11:38:49.097811 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.097793    2578 scope.go:117] "RemoveContainer" containerID="484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"
Sep 10 11:38:49.106028 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.106011    2578 scope.go:117] "RemoveContainer" containerID="a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"
Sep 10 11:38:49.106320 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:49.106299    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903\": container with ID starting with a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903 not found: ID does not exist" containerID="a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"
Sep 10 11:38:49.106404 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.106330    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903"} err="failed to get container status \"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903\": rpc error: code = NotFound desc = could not find container \"a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903\": container with ID starting with a85009952c4d57433a20a55682bf6cc2bf6f4e4c76a8fb978be677b30a2b3903 not found: ID does not exist"
Sep 10 11:38:49.106404 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.106347    2578 scope.go:117] "RemoveContainer" containerID="484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"
Sep 10 11:38:49.106586 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:38:49.106570    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95\": container with ID starting with 484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95 not found: ID does not exist" containerID="484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"
Sep 10 11:38:49.106629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.106591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95"} err="failed to get container status \"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95\": rpc error: code = NotFound desc = could not find container \"484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95\": container with ID starting with 484a9ac1664cbabb10a7ecf0290a37b3862e1406343a0dd395ff80b4b5168b95 not found: ID does not exist"
Sep 10 11:38:49.114156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.114061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:38:49.116132 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:49.116114    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7a970e-openshift-default-6f6c75fd7b-dp4s6"]
Sep 10 11:38:50.456404 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:50.456373    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" path="/var/lib/kubelet/pods/4b79e358-0209-4652-bd19-304a3fdeb0f4/volumes"
Sep 10 11:38:54.514351 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.514311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:54.628123 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.628075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:54.731728 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.731688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:54.825329 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.825234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"]
Sep 10 11:38:54.831326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.831296    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c2e44c-5mbzb"]
Sep 10 11:38:54.841051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.841009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:54.943899 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:54.943857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:55.242854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:55.242806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:38:56.456339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:38:56.456311    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1dbf712-b246-4244-b3a5-093ea417de12" path="/var/lib/kubelet/pods/b1dbf712-b246-4244-b3a5-093ea417de12/volumes"
Sep 10 11:39:02.253454 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:02.253399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:02.358472 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:02.358436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:02.543332 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:02.543246    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:39:02.543510 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:02.543470    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" containerID="cri-o://1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4" gracePeriod=30
Sep 10 11:39:02.597533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:02.597475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:03.170533 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:03.170494    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:39:03.170747 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:03.170572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:39:03.305900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:03.305866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:07.795902 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.795877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:39:07.926737 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.926949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.926949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkb9k\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.926949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.926949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.926949 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.927271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.926959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.927271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.927271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket\") pod \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\" (UID: \"9fc74c2e-7620-482b-be99-f33eb6eb5e58\") "
Sep 10 11:39:07.927271 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927077    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:07.927475 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data" (OuterVolumeSpecName: "istio-data") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:07.927475 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:39:07.927743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927596    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:07.927743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927619    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:07.927743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927633    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:07.927911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:07.927911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.927863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:07.929280 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.929249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token" (OuterVolumeSpecName: "istio-token") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:39:07.929394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.929295    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:07.929394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.929301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k" (OuterVolumeSpecName: "kube-api-access-bkb9k") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "kube-api-access-bkb9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:39:07.929394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:07.929370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9fc74c2e-7620-482b-be99-f33eb6eb5e58" (UID: "9fc74c2e-7620-482b-be99-f33eb6eb5e58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:39:08.028391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028356    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.028391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028385    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.028391 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028396    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkb9k\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-kube-api-access-bkb9k\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.028631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028405    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.028631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028413    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc74c2e-7620-482b-be99-f33eb6eb5e58-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.028631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.028421    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc74c2e-7620-482b-be99-f33eb6eb5e58-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:08.074886 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.074854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:08.199403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.199308    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerID="1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4" exitCode=0
Sep 10 11:39:08.199403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.199381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"
Sep 10 11:39:08.199403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.199395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerDied","Data":"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"}
Sep 10 11:39:08.199665 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.199474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5" event={"ID":"9fc74c2e-7620-482b-be99-f33eb6eb5e58","Type":"ContainerDied","Data":"dfabb65b8365000b437a13f5aef7152585ea8c1edd4090e18ca52866def860f1"}
Sep 10 11:39:08.199665 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.199499    2578 scope.go:117] "RemoveContainer" containerID="1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"
Sep 10 11:39:08.211358 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.211339    2578 scope.go:117] "RemoveContainer" containerID="b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"
Sep 10 11:39:08.219972 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.219956    2578 scope.go:117] "RemoveContainer" containerID="1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"
Sep 10 11:39:08.220278 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:39:08.220260    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4\": container with ID starting with 1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4 not found: ID does not exist" containerID="1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"
Sep 10 11:39:08.220350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.220285    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4"} err="failed to get container status \"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4\": rpc error: code = NotFound desc = could not find container \"1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4\": container with ID starting with 1b120401c67495a7aa0edbae11ca8562e35c3465dac0c3139a4866c15e5a3eb4 not found: ID does not exist"
Sep 10 11:39:08.220350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.220301    2578 scope.go:117] "RemoveContainer" containerID="b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"
Sep 10 11:39:08.220573 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:39:08.220555    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8\": container with ID starting with b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8 not found: ID does not exist" containerID="b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"
Sep 10 11:39:08.220625 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.220582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8"} err="failed to get container status \"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8\": rpc error: code = NotFound desc = could not find container \"b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8\": container with ID starting with b6cf0732b2b3e0fbd2b15ec4eacbefc7d3de3c0e2968e6e2d4d6f31adf716ea8 not found: ID does not exist"
Sep 10 11:39:08.230501 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.230472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:39:08.236988 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.236963    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c2e44c-openshift-default-5df7dc4648-px9q5"]
Sep 10 11:39:08.456595 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:08.456516    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" path="/var/lib/kubelet/pods/9fc74c2e-7620-482b-be99-f33eb6eb5e58/volumes"
Sep 10 11:39:09.140570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:09.140540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:09.589394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:09.589313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:10.006254 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006218    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:39:10.006682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006662    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:10.006682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006682    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006712    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006720    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c4c6ef-a8ae-4e56-932f-18ce66c3a2bf" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006746    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1dbf712-b246-4244-b3a5-093ea417de12" containerName="revoke-keys"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006754    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1dbf712-b246-4244-b3a5-093ea417de12" containerName="revoke-keys"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.006839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.007171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006866    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:10.007171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006878    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:10.007171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006889    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.007171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006899    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc74c2e-7620-482b-be99-f33eb6eb5e58" containerName="istio-proxy"
Sep 10 11:39:10.007171 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.006908    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1dbf712-b246-4244-b3a5-093ea417de12" containerName="revoke-keys"
Sep 10 11:39:10.011382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.011365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.016932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.016908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3f4d4b-openshift-default-dockercfg-gk4ld\""
Sep 10 11:39:10.082926 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.082870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:10.107810 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.107777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:39:10.148534 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867jf\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.148995 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.148957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250389 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867jf\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.250928 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.250903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.251250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.251174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.251250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.251224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.251405 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.251385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.253254 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.253227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.253449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.253434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.265917 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.265839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.271202 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.271181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867jf\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf\") pod \"e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") " pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.287480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.287452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:10.322756 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.322716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:10.616014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.615982    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:39:10.624345 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:39:10.624321    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24c7ac0_1b28_45bd_bca4_bf24b44d03a8.slice/crio-1fa56bd67b1f060c4a4c320eb5b8bb146ab17066ecb2e5d8cabf9abcd2a8f5a1 WatchSource:0}: Error finding container 1fa56bd67b1f060c4a4c320eb5b8bb146ab17066ecb2e5d8cabf9abcd2a8f5a1: Status 404 returned error can't find the container with id 1fa56bd67b1f060c4a4c320eb5b8bb146ab17066ecb2e5d8cabf9abcd2a8f5a1
Sep 10 11:39:10.626655 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.626620    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:10.626781 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.626687    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:10.626781 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:10.626720    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:11.221615 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:11.221580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerStarted","Data":"977b0c3ed6ef3c6045006c9dda1b08761dbf77ac78db72e1480c79037244b2bb"}
Sep 10 11:39:11.222015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:11.221634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerStarted","Data":"1fa56bd67b1f060c4a4c320eb5b8bb146ab17066ecb2e5d8cabf9abcd2a8f5a1"}
Sep 10 11:39:11.259343 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:11.259290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podStartSLOduration=2.2592717860000002 podStartE2EDuration="2.259271786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:39:11.254251968 +0000 UTC m=+1469.329577632" watchObservedRunningTime="2026-09-10 11:39:11.259271786 +0000 UTC m=+1469.334597507"
Sep 10 11:39:11.323041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:11.323000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:11.328192 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:11.328164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:12.227012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:12.226977    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:12.228078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:12.228056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:13.288170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:13.288131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:13.425370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:13.425334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:13.796994 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:13.796964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:13.895733 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:13.895700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:14.032564 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:14.032524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:14.143511 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:14.143481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:14.463506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:14.463404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:14.592832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:14.592799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:14.712773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:14.712738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:15.478240 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:15.478210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:15.610349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:15.610320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:22.388520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:22.388481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:22.492072 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:22.492032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:22.673097 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:22.672993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:39:22.708014 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:22.707978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:23.286881 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:23.286832    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy" containerID="cri-o://ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38" gracePeriod=30
Sep 10 11:39:25.578326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:25.578295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:28.217310 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.217279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:28.552450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.552421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:39:28.636182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636182 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636271    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97v95\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636449 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs\") pod \"739e6cb2-8d8a-46c5-8418-48987394ea10\" (UID: \"739e6cb2-8d8a-46c5-8418-48987394ea10\") "
Sep 10 11:39:28.636777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data" (OuterVolumeSpecName: "istio-data") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:28.636777 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:28.636888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636779    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.636888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:28.636981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.636936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:39:28.637026 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.637004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:28.638810 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.638763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:39:28.638985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.638874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token" (OuterVolumeSpecName: "istio-token") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:39:28.638985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.638953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95" (OuterVolumeSpecName: "kube-api-access-97v95") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "kube-api-access-97v95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:39:28.639333 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.639311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "739e6cb2-8d8a-46c5-8418-48987394ea10" (UID: "739e6cb2-8d8a-46c5-8418-48987394ea10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:39:28.737303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737263    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737296    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737305    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737303 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737314    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737322    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737330    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97v95\" (UniqueName: \"kubernetes.io/projected/739e6cb2-8d8a-46c5-8418-48987394ea10-kube-api-access-97v95\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737338    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739e6cb2-8d8a-46c5-8418-48987394ea10-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:28.737570 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:28.737346    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739e6cb2-8d8a-46c5-8418-48987394ea10-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:39:29.330232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.330193    2578 generic.go:356] "Generic (PLEG): container finished" podID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerID="ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38" exitCode=0
Sep 10 11:39:29.330647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.330247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"
Sep 10 11:39:29.330647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.330276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerDied","Data":"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"}
Sep 10 11:39:29.330647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.330336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq" event={"ID":"739e6cb2-8d8a-46c5-8418-48987394ea10","Type":"ContainerDied","Data":"a6795b186f70aa2f3d7dfc0c53052305955d63fc4af49fe03c17e1f6e2351d3c"}
Sep 10 11:39:29.330647 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.330359    2578 scope.go:117] "RemoveContainer" containerID="ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"
Sep 10 11:39:29.339765 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.339746    2578 scope.go:117] "RemoveContainer" containerID="46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"
Sep 10 11:39:29.348308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.348290    2578 scope.go:117] "RemoveContainer" containerID="ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"
Sep 10 11:39:29.348611 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:39:29.348592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38\": container with ID starting with ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38 not found: ID does not exist" containerID="ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"
Sep 10 11:39:29.348660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.348620    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38"} err="failed to get container status \"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38\": rpc error: code = NotFound desc = could not find container \"ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38\": container with ID starting with ed421ebf5f38fca3be36fc20982b093c9d52a6221ea23087c3a00312223d8c38 not found: ID does not exist"
Sep 10 11:39:29.348660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.348638    2578 scope.go:117] "RemoveContainer" containerID="46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"
Sep 10 11:39:29.348890 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:39:29.348873    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953\": container with ID starting with 46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953 not found: ID does not exist" containerID="46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"
Sep 10 11:39:29.348932 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.348895    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953"} err="failed to get container status \"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953\": rpc error: code = NotFound desc = could not find container \"46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953\": container with ID starting with 46cc77b8004ec6c0f62ff0a2dca1e931b95b565e4c4a9de58dc209534d13f953 not found: ID does not exist"
Sep 10 11:39:29.353878 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.353854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:39:29.359502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:29.359482    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-282355-openshift-default-84b9cb468b-hlfnq"]
Sep 10 11:39:30.455413 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:30.455379    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" path="/var/lib/kubelet/pods/739e6cb2-8d8a-46c5-8418-48987394ea10/volumes"
Sep 10 11:39:31.351048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.351016    2578 generic.go:356] "Generic (PLEG): container finished" podID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerID="977b0c3ed6ef3c6045006c9dda1b08761dbf77ac78db72e1480c79037244b2bb" exitCode=0
Sep 10 11:39:31.351279 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.351123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerDied","Data":"977b0c3ed6ef3c6045006c9dda1b08761dbf77ac78db72e1480c79037244b2bb"}
Sep 10 11:39:31.351581 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.351564    2578 scope.go:117] "RemoveContainer" containerID="977b0c3ed6ef3c6045006c9dda1b08761dbf77ac78db72e1480c79037244b2bb"
Sep 10 11:39:31.352670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.352643    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:31.352724 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.352708    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:31.352760 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:31.352736    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:32.359113 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:32.359055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerStarted","Data":"bda078c783883ba691739818580f1ac38b04eae77c65f3a224b61d386da59571"}
Sep 10 11:39:33.323423 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:33.323385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:33.323629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:33.323607    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 11:39:33.323694 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:33.323673    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 11:39:34.323393 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:34.323354    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 11:39:34.323798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:34.323416    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 11:39:35.323993 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:35.323955    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 11:39:35.324425 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:35.324016    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 11:39:36.050785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:36.050728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:36.323531 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:36.323422    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 11:39:36.323531 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:36.323491    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 11:39:37.324140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:37.324073    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 11:39:37.324554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:37.324161    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 11:39:38.327685 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:38.327654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:38.328194 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:38.327915    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:38.328755 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:38.328736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:39:38.336614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:38.336591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:39.094355 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.094322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:39.522311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522273    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522631    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522638    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b79e358-0209-4652-bd19-304a3fdeb0f4" containerName="istio-proxy"
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522663    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:39:39.522695 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522669    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:39:39.522943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.522719    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:39:39.525861 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.525840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.531896 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.531868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-420024-openshift-default-dockercfg-bq9zc\""
Sep 10 11:39:39.558923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.558881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:39:39.625383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.625300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:39.638798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.638765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.638990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.638831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.638990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.638912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.638990 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.638970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.639155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.638995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.639155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.639012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.639155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.639030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.639155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.639142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.639304 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.639158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sl7l\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740344 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740550 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sl7l\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.740913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.741022 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.740947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.741142 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.741080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.741269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.741179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.741269 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.741187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.743450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.743430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.743589 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.743574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.754023 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.754001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.756690 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.756665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sl7l\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l\") pod \"e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") " pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.835996 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.835901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:39.874801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:39.874761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:40.132380 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.132350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:39:40.137142 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:39:40.137118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e78459_1e55_4479_b9e5_0dd69185a64c.slice/crio-22b403118e584d6af4818cfcdba5f1842e9832161f1d456907584ca90389dc4d WatchSource:0}: Error finding container 22b403118e584d6af4818cfcdba5f1842e9832161f1d456907584ca90389dc4d: Status 404 returned error can't find the container with id 22b403118e584d6af4818cfcdba5f1842e9832161f1d456907584ca90389dc4d
Sep 10 11:39:40.139326 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.139295    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:40.139392 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.139361    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:40.139429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.139389    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:39:40.403010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.402912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerStarted","Data":"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"}
Sep 10 11:39:40.403010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.402974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerStarted","Data":"22b403118e584d6af4818cfcdba5f1842e9832161f1d456907584ca90389dc4d"}
Sep 10 11:39:40.454790 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.454715    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podStartSLOduration=1.454689226 podStartE2EDuration="1.454689226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:39:40.451610283 +0000 UTC m=+1498.526935948" watchObservedRunningTime="2026-09-10 11:39:40.454689226 +0000 UTC m=+1498.530014891"
Sep 10 11:39:40.836635 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:40.836530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:41.841959 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:41.841921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:42.416012 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.415979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:42.417050 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.417027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:39:42.505198 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.505165    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:39:42.506660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.506638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:39:42.757381 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.757286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:42.891807 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:42.891767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:43.949321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:43.949288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:44.051024 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:44.050979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:44.582532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:44.582482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:44.681512 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:44.681477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:39:58.458262 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:58.458217    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 11:39:58.458800 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:39:58.458277    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 11:40:00.514018 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.513985    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerID="ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a" exitCode=0
Sep 10 11:40:00.514466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.514056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerDied","Data":"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"}
Sep 10 11:40:00.514466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.514462    2578 scope.go:117] "RemoveContainer" containerID="ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"
Sep 10 11:40:00.515568 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.515540    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:00.515631 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.515611    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:00.515667 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:00.515642    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:01.522455 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:01.522425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerStarted","Data":"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"}
Sep 10 11:40:01.836524 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:01.836421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:40:01.836743 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:01.836722    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 11:40:01.836793 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:01.836773    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 11:40:02.836924 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:02.836887    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 11:40:02.837429 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:02.836943    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 11:40:03.837252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:03.837175    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 11:40:03.837252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:03.837236    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 11:40:04.837247 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:04.837209    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 11:40:04.837666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:04.837269    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 11:40:05.836337 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:05.836298    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 11:40:05.836532 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:05.836367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 11:40:06.839808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:06.839780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:40:06.840250 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:06.840127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:40:06.840738 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:06.840722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:40:08.543213 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:08.543180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:18.998327 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:18.998273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:44.188201 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.188148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:44.506156 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.506059    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:40:44.506760 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.506729    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="739e6cb2-8d8a-46c5-8418-48987394ea10" containerName="istio-proxy"
Sep 10 11:40:44.509956 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.509935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.512908 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.512883    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-34501f-openshift-default-dockercfg-cvz6s\""
Sep 10 11:40:44.526614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.526588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:40:44.576815 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.576759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:44.580492 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xpd\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580561 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580714 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580877 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580913 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.580948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.580916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.682510 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.682510 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.682783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.682783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.682783 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xpd\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.682989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683019 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.683009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.683205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683382 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.683349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.683483 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.683463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.685370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.685349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.685834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.685812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.697830 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.697795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xpd\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.705915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.705886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token\") pod \"e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") " pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:44.710319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.710276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:44.820258 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:44.820160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:45.101939 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.101911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:40:45.103741 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:40:45.103712    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf898b53_2a29_4c6b_a333_7f4939db47cf.slice/crio-b477faa4e6ce14611f656097cab8466830b1cb922a29fd05d82795a4aa48de4f WatchSource:0}: Error finding container b477faa4e6ce14611f656097cab8466830b1cb922a29fd05d82795a4aa48de4f: Status 404 returned error can't find the container with id b477faa4e6ce14611f656097cab8466830b1cb922a29fd05d82795a4aa48de4f
Sep 10 11:40:45.105991 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.105953    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:45.106116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.106037    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:45.106116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.106080    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:40:45.761331 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.761293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerStarted","Data":"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"}
Sep 10 11:40:45.761773 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.761347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerStarted","Data":"b477faa4e6ce14611f656097cab8466830b1cb922a29fd05d82795a4aa48de4f"}
Sep 10 11:40:45.790717 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.790657    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podStartSLOduration=1.7906399849999999 podStartE2EDuration="1.790639985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:40:45.78766227 +0000 UTC m=+1563.862987934" watchObservedRunningTime="2026-09-10 11:40:45.790639985 +0000 UTC m=+1563.865965648"
Sep 10 11:40:45.820596 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:45.820552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:46.826041 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:46.826008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:47.738030 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:47.737989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:47.771476 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:47.771445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:47.772834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:47.772818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:40:47.843420 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:47.843392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.062958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.062926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.234225 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.234192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.396013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.395981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.608970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.608939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.704122 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.704011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:49.802007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.801973    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:40:49.805572 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.805548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:49.811959 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.811937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-34501f-dockercfg-2tr4l\""
Sep 10 11:40:49.812456 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.812437    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-34501f\""
Sep 10 11:40:49.825682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.825654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:40:49.938211 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.938174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrj7\" (UniqueName: \"kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:49.938377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.938338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:49.977285 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:49.977202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:40:50.039513 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.039466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.039729 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.039584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrj7\" (UniqueName: \"kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.040206 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.040184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.049901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.049878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrj7\" (UniqueName: \"kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7\") pod \"payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.116768 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.116732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.396025 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.395998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:40:50.398142 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:40:50.398114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8622ddb5_3434_45a9_9d3d_3f3b6177d913.slice/crio-b984a33d1dbf0ad8c1ecaa76632b7acfb9bce6d14b29577fd902238dc2b752be WatchSource:0}: Error finding container b984a33d1dbf0ad8c1ecaa76632b7acfb9bce6d14b29577fd902238dc2b752be: Status 404 returned error can't find the container with id b984a33d1dbf0ad8c1ecaa76632b7acfb9bce6d14b29577fd902238dc2b752be
Sep 10 11:40:50.789769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.789679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" event={"ID":"8622ddb5-3434-45a9-9d3d-3f3b6177d913","Type":"ContainerStarted","Data":"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"}
Sep 10 11:40:50.789769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.789746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:40:50.789954 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.789772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" event={"ID":"8622ddb5-3434-45a9-9d3d-3f3b6177d913","Type":"ContainerStarted","Data":"b984a33d1dbf0ad8c1ecaa76632b7acfb9bce6d14b29577fd902238dc2b752be"}
Sep 10 11:40:50.811610 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:40:50.811553    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" podStartSLOduration=1.811533491 podStartE2EDuration="1.811533491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:40:50.809837052 +0000 UTC m=+1568.885162741" watchObservedRunningTime="2026-09-10 11:40:50.811533491 +0000 UTC m=+1568.886859155"
Sep 10 11:41:01.799319 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:01.799277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:41:07.888678 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.888642    2578 generic.go:356] "Generic (PLEG): container finished" podID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerID="659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f" exitCode=0
Sep 10 11:41:07.889119 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.888716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerDied","Data":"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"}
Sep 10 11:41:07.889167 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.889152    2578 scope.go:117] "RemoveContainer" containerID="659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"
Sep 10 11:41:07.890286 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.890249    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:41:07.890400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.890321    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:41:07.890400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:07.890353    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 11:41:08.900529 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:08.900480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerStarted","Data":"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"}
Sep 10 11:41:09.820915 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:09.820873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:09.821170 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:09.821146    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 11:41:09.821231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:09.821208    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 11:41:10.820805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:10.820769    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 11:41:10.821231 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:10.820825    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 11:41:11.821471 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:11.821431    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 11:41:11.821882 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:11.821492    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 11:41:12.820666 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:12.820630    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 11:41:12.820875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:12.820679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:12.820875 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:12.820713    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 11:41:13.821523 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:13.821484    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 11:41:13.821923 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:13.821550    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 11:41:14.272302 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:14.272260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:14.825311 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:14.825282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:14.825734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:14.825711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:14.826416 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:14.826396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:19.849212 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:19.849163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:19.896395 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:19.896363    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:41:19.896638 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:19.896611    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" podUID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" containerName="payload-pre-processing" containerID="cri-o://8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887" gracePeriod=30
Sep 10 11:41:19.934347 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:19.934317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:20.034216 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.034178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:20.134753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.134728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:20.148162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.148135    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:41:20.248946 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.248912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjrj7\" (UniqueName: \"kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7\") pod \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") "
Sep 10 11:41:20.249163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.249059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume\") pod \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\" (UID: \"8622ddb5-3434-45a9-9d3d-3f3b6177d913\") "
Sep 10 11:41:20.249466 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.249424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8622ddb5-3434-45a9-9d3d-3f3b6177d913" (UID: "8622ddb5-3434-45a9-9d3d-3f3b6177d913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:41:20.251509 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.251481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7" (OuterVolumeSpecName: "kube-api-access-bjrj7") pod "8622ddb5-3434-45a9-9d3d-3f3b6177d913" (UID: "8622ddb5-3434-45a9-9d3d-3f3b6177d913"). InnerVolumeSpecName "kube-api-access-bjrj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:41:20.350446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.350411    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrj7\" (UniqueName: \"kubernetes.io/projected/8622ddb5-3434-45a9-9d3d-3f3b6177d913-kube-api-access-bjrj7\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:20.350446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.350439    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8622ddb5-3434-45a9-9d3d-3f3b6177d913-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:20.475804 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.475774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:20.971174 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.971138    2578 generic.go:356] "Generic (PLEG): container finished" podID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" containerID="8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887" exitCode=0
Sep 10 11:41:20.971607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.971205    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"
Sep 10 11:41:20.971607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.971229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" event={"ID":"8622ddb5-3434-45a9-9d3d-3f3b6177d913","Type":"ContainerDied","Data":"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"}
Sep 10 11:41:20.971607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.971305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww" event={"ID":"8622ddb5-3434-45a9-9d3d-3f3b6177d913","Type":"ContainerDied","Data":"b984a33d1dbf0ad8c1ecaa76632b7acfb9bce6d14b29577fd902238dc2b752be"}
Sep 10 11:41:20.971607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.971328    2578 scope.go:117] "RemoveContainer" containerID="8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"
Sep 10 11:41:20.980731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.980708    2578 scope.go:117] "RemoveContainer" containerID="8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"
Sep 10 11:41:20.981020 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:41:20.981001    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887\": container with ID starting with 8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887 not found: ID does not exist" containerID="8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"
Sep 10 11:41:20.981116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.981031    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887"} err="failed to get container status \"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887\": rpc error: code = NotFound desc = could not find container \"8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887\": container with ID starting with 8ac0eb831cced875d64daa17f22c5834883d1e6e2d2484b8bcf77c623dbda887 not found: ID does not exist"
Sep 10 11:41:20.991626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.991601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:41:20.995342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:20.995320    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-34501f-797d748b98-sfqww"]
Sep 10 11:41:22.455620 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:22.455585    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" path="/var/lib/kubelet/pods/8622ddb5-3434-45a9-9d3d-3f3b6177d913/volumes"
Sep 10 11:41:26.801057 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:26.801026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:26.902128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:26.902078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:27.075431 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:27.075343    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:41:27.075597 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:27.075535    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" containerID="cri-o://0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6" gracePeriod=30
Sep 10 11:41:27.094470 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:27.094434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:29.828204 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:29.828161    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:41:29.828661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:29.828247    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:41:31.101911 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:31.101876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:32.322438 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.322413    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:32.473940 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.473902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.473955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.473981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56xpd\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474183 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474539 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474226    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert\") pod \"cf898b53-2a29-4c6b-a333-7f4939db47cf\" (UID: \"cf898b53-2a29-4c6b-a333-7f4939db47cf\") "
Sep 10 11:41:32.474592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:41:32.474592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474336    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:41:32.474592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:41:32.474764 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:41:32.474832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.474814    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:41:32.476473 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.476441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:41:32.476594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.476516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd" (OuterVolumeSpecName: "kube-api-access-56xpd") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "kube-api-access-56xpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:41:32.476594 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.476526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:41:32.476735 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.476720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf898b53-2a29-4c6b-a333-7f4939db47cf" (UID: "cf898b53-2a29-4c6b-a333-7f4939db47cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:41:32.575787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575750    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.575787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575780    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.575787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575790    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.575787 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575798    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.576193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575809    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xpd\" (UniqueName: \"kubernetes.io/projected/cf898b53-2a29-4c6b-a333-7f4939db47cf-kube-api-access-56xpd\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.576193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575824    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.576193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575837    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.576193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575849    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf898b53-2a29-4c6b-a333-7f4939db47cf-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:32.576193 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:32.575860    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf898b53-2a29-4c6b-a333-7f4939db47cf-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:41:33.041818 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.041787    2578 generic.go:356] "Generic (PLEG): container finished" podID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerID="0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6" exitCode=0
Sep 10 11:41:33.042015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.041839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"
Sep 10 11:41:33.042015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.041879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerDied","Data":"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"}
Sep 10 11:41:33.042015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.041933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4" event={"ID":"cf898b53-2a29-4c6b-a333-7f4939db47cf","Type":"ContainerDied","Data":"b477faa4e6ce14611f656097cab8466830b1cb922a29fd05d82795a4aa48de4f"}
Sep 10 11:41:33.042015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.041954    2578 scope.go:117] "RemoveContainer" containerID="0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"
Sep 10 11:41:33.052384 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.052365    2578 scope.go:117] "RemoveContainer" containerID="659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"
Sep 10 11:41:33.061843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.061825    2578 scope.go:117] "RemoveContainer" containerID="0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"
Sep 10 11:41:33.062154 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:41:33.062127    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6\": container with ID starting with 0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6 not found: ID does not exist" containerID="0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"
Sep 10 11:41:33.062245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.062160    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6"} err="failed to get container status \"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6\": rpc error: code = NotFound desc = could not find container \"0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6\": container with ID starting with 0377b8f8845f6e167f403492f416fff02589d6a9611f568708b3a9946d6ac0f6 not found: ID does not exist"
Sep 10 11:41:33.062245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.062186    2578 scope.go:117] "RemoveContainer" containerID="659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"
Sep 10 11:41:33.062456 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:41:33.062437    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f\": container with ID starting with 659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f not found: ID does not exist" containerID="659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"
Sep 10 11:41:33.062502 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.062463    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f"} err="failed to get container status \"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f\": rpc error: code = NotFound desc = could not find container \"659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f\": container with ID starting with 659a3e7607559f19a572d9c40a18a4733b956c7dcf3ddbfe4c2ce351a68f3b9f not found: ID does not exist"
Sep 10 11:41:33.067736 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.067716    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:41:33.073609 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:33.073588    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34501f-openshift-default-575f67df7d-64vx4"]
Sep 10 11:41:34.456607 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:34.456573    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" path="/var/lib/kubelet/pods/cf898b53-2a29-4c6b-a333-7f4939db47cf/volumes"
Sep 10 11:41:42.781555 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:42.781522    2578 scope.go:117] "RemoveContainer" containerID="5e71e62dbe6461cb712702e856f7ed80541fcaeb22bcf25e01145826ae5c374b"
Sep 10 11:41:46.528531 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:46.528480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:47.982015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:47.981963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:41:52.224514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:41:52.224456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:28.615983 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:28.615953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:29.088152 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:29.088056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:29.922354 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:29.922308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:35.858507 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:35.858458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:35.962621 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:35.962584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:36.162798 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:36.162699    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:42:36.162985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:36.162963    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" containerID="cri-o://bda078c783883ba691739818580f1ac38b04eae77c65f3a224b61d386da59571" gracePeriod=30
Sep 10 11:42:36.212362 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:36.212328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:36.669374 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:36.669341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:38.329855 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:38.329818    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 11:42:38.330252 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:38.329883    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 11:42:39.587658 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:39.587603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:39.669660 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:39.669632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:41.398834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.398798    2578 generic.go:356] "Generic (PLEG): container finished" podID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerID="bda078c783883ba691739818580f1ac38b04eae77c65f3a224b61d386da59571" exitCode=0
Sep 10 11:42:41.399315 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.398844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerDied","Data":"bda078c783883ba691739818580f1ac38b04eae77c65f3a224b61d386da59571"}
Sep 10 11:42:41.399315 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.398948    2578 scope.go:117] "RemoveContainer" containerID="977b0c3ed6ef3c6045006c9dda1b08761dbf77ac78db72e1480c79037244b2bb"
Sep 10 11:42:41.424051 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.424025    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:42:41.535812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.535812 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.535976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536047 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536066    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867jf\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf\") pod \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\" (UID: \"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8\") "
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:42:41.536370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536546    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536566    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536581    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536594    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.536664 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.536606    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.538548 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.538527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf" (OuterVolumeSpecName: "kube-api-access-867jf") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "kube-api-access-867jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:42:41.538652 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.538604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:42:41.538916 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.538900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:42:41.538970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.538914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" (UID: "f24c7ac0-1b28-45bd-bca4-bf24b44d03a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:42:41.637813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.637778    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.637813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.637807    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.637813 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.637819    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:41.638049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:41.637833    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867jf\" (UniqueName: \"kubernetes.io/projected/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8-kube-api-access-867jf\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:42:42.406703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:42.406677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"
Sep 10 11:42:42.406703 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:42.406686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8" event={"ID":"f24c7ac0-1b28-45bd-bca4-bf24b44d03a8","Type":"ContainerDied","Data":"1fa56bd67b1f060c4a4c320eb5b8bb146ab17066ecb2e5d8cabf9abcd2a8f5a1"}
Sep 10 11:42:42.407235 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:42.406761    2578 scope.go:117] "RemoveContainer" containerID="bda078c783883ba691739818580f1ac38b04eae77c65f3a224b61d386da59571"
Sep 10 11:42:42.456063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:42.456036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:42:42.456063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:42.456070    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f4d4b-openshift-default-6944947bcb-lhng8"]
Sep 10 11:42:44.455616 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:44.455586    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" path="/var/lib/kubelet/pods/f24c7ac0-1b28-45bd-bca4-bf24b44d03a8/volumes"
Sep 10 11:42:47.267771 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:47.267738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:47.368951 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:47.368916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:47.466505 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:47.466470    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:47.581930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:47.581847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:47.888648 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:47.888612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:54.761592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:54.761551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:54.836120 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:54.836071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:55.032981 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:55.032880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:42:55.033169 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:55.033132    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy" containerID="cri-o://d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5" gracePeriod=30
Sep 10 11:42:55.053711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:55.053679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:55.533662 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:55.533625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:42:57.082556 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:42:57.082517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:00.288694 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.288666    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:43:00.418537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sl7l\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418537 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.418826 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418795    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token\") pod \"d9e78459-1e55-4479-b9e5-0dd69185a64c\" (UID: \"d9e78459-1e55-4479-b9e5-0dd69185a64c\") "
Sep 10 11:43:00.419236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:00.419236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.418949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:00.419236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:43:00.419236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:00.419236 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:00.419670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419220    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.419670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419261    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e78459-1e55-4479-b9e5-0dd69185a64c-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.419670 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.419360    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.420958 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.420931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l" (OuterVolumeSpecName: "kube-api-access-4sl7l") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "kube-api-access-4sl7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:00.421073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.420961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:00.421073 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.421042    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:00.421362 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.421345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9e78459-1e55-4479-b9e5-0dd69185a64c" (UID: "d9e78459-1e55-4479-b9e5-0dd69185a64c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:43:00.502521 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.502488    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerID="d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5" exitCode=0
Sep 10 11:43:00.502692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.502545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"
Sep 10 11:43:00.502692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.502571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerDied","Data":"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"}
Sep 10 11:43:00.502692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.502630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql" event={"ID":"d9e78459-1e55-4479-b9e5-0dd69185a64c","Type":"ContainerDied","Data":"22b403118e584d6af4818cfcdba5f1842e9832161f1d456907584ca90389dc4d"}
Sep 10 11:43:00.502692 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.502656    2578 scope.go:117] "RemoveContainer" containerID="d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"
Sep 10 11:43:00.512044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.512017    2578 scope.go:117] "RemoveContainer" containerID="ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"
Sep 10 11:43:00.520488 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520472    2578 scope.go:117] "RemoveContainer" containerID="d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"
Sep 10 11:43:00.520576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520558    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520582    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520598    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sl7l\" (UniqueName: \"kubernetes.io/projected/d9e78459-1e55-4479-b9e5-0dd69185a64c-kube-api-access-4sl7l\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520637 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520628    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520643    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e78459-1e55-4479-b9e5-0dd69185a64c-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:00.520820 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:00.520773    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5\": container with ID starting with d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5 not found: ID does not exist" containerID="d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"
Sep 10 11:43:00.520820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520797    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5"} err="failed to get container status \"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5\": rpc error: code = NotFound desc = could not find container \"d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5\": container with ID starting with d12f729ffe2c9844d8f57b26dd7dfbfdc3fdc3f1d0f87f824c3482938eadc5a5 not found: ID does not exist"
Sep 10 11:43:00.520820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.520814    2578 scope.go:117] "RemoveContainer" containerID="ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"
Sep 10 11:43:00.521100 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:00.521057    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a\": container with ID starting with ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a not found: ID does not exist" containerID="ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"
Sep 10 11:43:00.521157 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.521119    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a"} err="failed to get container status \"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a\": rpc error: code = NotFound desc = could not find container \"ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a\": container with ID starting with ec33387dcb20c2349e277e8d6389d492eab4e3f169b8a5450837fd8f8fed304a not found: ID does not exist"
Sep 10 11:43:00.531506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.531483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:43:00.536074 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:00.536054    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-420024-openshift-default-67c98f74f6-vvbql"]
Sep 10 11:43:02.456172 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:02.456139    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" path="/var/lib/kubelet/pods/d9e78459-1e55-4479-b9e5-0dd69185a64c/volumes"
Sep 10 11:43:05.632999 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:05.632912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:06.390297 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.390262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:06.424558 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.424522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:43:06.424846 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.424817    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" podUID="393b3251-99be-4684-b1b0-99643174fae0" containerName="payload-processing" containerID="cri-o://93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13" gracePeriod=30
Sep 10 11:43:06.485794 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.485763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:06.585809 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.585774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:06.663572 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.663492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:06.685070 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.685046    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:43:06.770965 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.770928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqs2q\" (UniqueName: \"kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q\") pod \"393b3251-99be-4684-b1b0-99643174fae0\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") "
Sep 10 11:43:06.771186 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.770996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume\") pod \"393b3251-99be-4684-b1b0-99643174fae0\" (UID: \"393b3251-99be-4684-b1b0-99643174fae0\") "
Sep 10 11:43:06.771394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.771367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "393b3251-99be-4684-b1b0-99643174fae0" (UID: "393b3251-99be-4684-b1b0-99643174fae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:43:06.773312 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.773279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q" (OuterVolumeSpecName: "kube-api-access-zqs2q") pod "393b3251-99be-4684-b1b0-99643174fae0" (UID: "393b3251-99be-4684-b1b0-99643174fae0"). InnerVolumeSpecName "kube-api-access-zqs2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:06.871675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.871640    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqs2q\" (UniqueName: \"kubernetes.io/projected/393b3251-99be-4684-b1b0-99643174fae0-kube-api-access-zqs2q\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:06.871675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:06.871669    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393b3251-99be-4684-b1b0-99643174fae0-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:07.046788 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.046687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:07.540394 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.540359    2578 generic.go:356] "Generic (PLEG): container finished" podID="393b3251-99be-4684-b1b0-99643174fae0" containerID="93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13" exitCode=0
Sep 10 11:43:07.540576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.540394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" event={"ID":"393b3251-99be-4684-b1b0-99643174fae0","Type":"ContainerDied","Data":"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"}
Sep 10 11:43:07.540576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.540449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt" event={"ID":"393b3251-99be-4684-b1b0-99643174fae0","Type":"ContainerDied","Data":"9d804f0cfb554c460b81cc5c1d3b13a8593e3f834a33bb5b2485fd9f24641687"}
Sep 10 11:43:07.540576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.540469    2578 scope.go:117] "RemoveContainer" containerID="93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"
Sep 10 11:43:07.540576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.540409    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"
Sep 10 11:43:07.549490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.549473    2578 scope.go:117] "RemoveContainer" containerID="93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"
Sep 10 11:43:07.549743 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:07.549725    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13\": container with ID starting with 93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13 not found: ID does not exist" containerID="93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"
Sep 10 11:43:07.549814 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.549749    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13"} err="failed to get container status \"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13\": rpc error: code = NotFound desc = could not find container \"93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13\": container with ID starting with 93f686d8434db64026e912e5217fa022c4f040035df3ae3777942fbfceb49c13 not found: ID does not exist"
Sep 10 11:43:07.565196 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.565172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:43:07.572372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:07.572349    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fde2a-589cd478f7-2psdt"]
Sep 10 11:43:08.455878 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:08.455846    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393b3251-99be-4684-b1b0-99643174fae0" path="/var/lib/kubelet/pods/393b3251-99be-4684-b1b0-99643174fae0/volumes"
Sep 10 11:43:13.497330 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:13.497299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:13.585916 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:13.585885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:13.768143 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:13.768031    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:43:13.768370 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:13.768315    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy" containerID="cri-o://2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e" gracePeriod=30
Sep 10 11:43:13.805541 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:13.805503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:14.252208 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:14.252172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:19.021395 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.021366    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:43:19.062633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wwm5\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062633 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.062901 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token\") pod \"7c215354-a78c-4cee-988f-b632ce0b27d3\" (UID: \"7c215354-a78c-4cee-988f-b632ce0b27d3\") "
Sep 10 11:43:19.063256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.062987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:19.063256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:19.063256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063196    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:19.063256 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:43:19.063490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063316    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.063490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063336    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.063490 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063349    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.063657 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.063504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:19.065495 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.065469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:43:19.065646 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.065613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:19.065748 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.065688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5" (OuterVolumeSpecName: "kube-api-access-2wwm5") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "kube-api-access-2wwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:19.065865 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.065842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c215354-a78c-4cee-988f-b632ce0b27d3" (UID: "7c215354-a78c-4cee-988f-b632ce0b27d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:19.164584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164540    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.164584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164576    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.164584 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164589    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwm5\" (UniqueName: \"kubernetes.io/projected/7c215354-a78c-4cee-988f-b632ce0b27d3-kube-api-access-2wwm5\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.164839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164603    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c215354-a78c-4cee-988f-b632ce0b27d3-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.164839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164614    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.164839 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.164624    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c215354-a78c-4cee-988f-b632ce0b27d3-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:19.602969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.602938    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerID="2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e" exitCode=0
Sep 10 11:43:19.603162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.602993    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"
Sep 10 11:43:19.603162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.603023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerDied","Data":"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"}
Sep 10 11:43:19.603162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.603107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b" event={"ID":"7c215354-a78c-4cee-988f-b632ce0b27d3","Type":"ContainerDied","Data":"d9e6ba041e5bd3b935fd37c4409b1ee122988c948b6ca0528b39260232c52f11"}
Sep 10 11:43:19.603162 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.603137    2578 scope.go:117] "RemoveContainer" containerID="2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"
Sep 10 11:43:19.612967 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.612948    2578 scope.go:117] "RemoveContainer" containerID="b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"
Sep 10 11:43:19.621440 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.621422    2578 scope.go:117] "RemoveContainer" containerID="2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"
Sep 10 11:43:19.621729 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:19.621710    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e\": container with ID starting with 2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e not found: ID does not exist" containerID="2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"
Sep 10 11:43:19.621808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.621743    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e"} err="failed to get container status \"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e\": rpc error: code = NotFound desc = could not find container \"2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e\": container with ID starting with 2fdeb782d77030263fe05d03491f34123335d70fc49bb37c28feb8abd730fd2e not found: ID does not exist"
Sep 10 11:43:19.621808 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.621769    2578 scope.go:117] "RemoveContainer" containerID="b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"
Sep 10 11:43:19.622036 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:19.622019    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee\": container with ID starting with b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee not found: ID does not exist" containerID="b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"
Sep 10 11:43:19.622173 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.622042    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee"} err="failed to get container status \"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee\": rpc error: code = NotFound desc = could not find container \"b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee\": container with ID starting with b84310f2765d42114c1102bb21602e6643f7d6fbc42a09f651378505ce0102ee not found: ID does not exist"
Sep 10 11:43:19.627457 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.627435    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:43:19.635000 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:19.634977    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fde2a-openshift-default-9f7ffd45d-lsz2b"]
Sep 10 11:43:20.456976 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:20.456945    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" path="/var/lib/kubelet/pods/7c215354-a78c-4cee-988f-b632ce0b27d3/volumes"
Sep 10 11:43:24.705843 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.705799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:24.742363 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.742330    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:43:24.742614 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.742588    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" podUID="647cc619-9dfd-4913-9f78-368bc5de76e9" containerName="payload-pre-processing" containerID="cri-o://f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2" gracePeriod=30
Sep 10 11:43:24.801464 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:24.801433    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647cc619_9dfd_4913_9f78_368bc5de76e9.slice/crio-conmon-f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 11:43:24.804006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.803971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:24.903361 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.903290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:24.997412 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:24.997389    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:43:25.005099 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.005061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:25.015138 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.015118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2qsp\" (UniqueName: \"kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp\") pod \"647cc619-9dfd-4913-9f78-368bc5de76e9\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") "
Sep 10 11:43:25.015226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.015214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume\") pod \"647cc619-9dfd-4913-9f78-368bc5de76e9\" (UID: \"647cc619-9dfd-4913-9f78-368bc5de76e9\") "
Sep 10 11:43:25.015536 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.015516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "647cc619-9dfd-4913-9f78-368bc5de76e9" (UID: "647cc619-9dfd-4913-9f78-368bc5de76e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:43:25.017445 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.017423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp" (OuterVolumeSpecName: "kube-api-access-s2qsp") pod "647cc619-9dfd-4913-9f78-368bc5de76e9" (UID: "647cc619-9dfd-4913-9f78-368bc5de76e9"). InnerVolumeSpecName "kube-api-access-s2qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:25.116422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.116376    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647cc619-9dfd-4913-9f78-368bc5de76e9-plugins-config-volume\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:25.116422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.116416    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qsp\" (UniqueName: \"kubernetes.io/projected/647cc619-9dfd-4913-9f78-368bc5de76e9-kube-api-access-s2qsp\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:25.353163 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.353049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:25.635168 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.635131    2578 generic.go:356] "Generic (PLEG): container finished" podID="647cc619-9dfd-4913-9f78-368bc5de76e9" containerID="f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2" exitCode=0
Sep 10 11:43:25.635349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.635177    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"
Sep 10 11:43:25.635349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.635219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" event={"ID":"647cc619-9dfd-4913-9f78-368bc5de76e9","Type":"ContainerDied","Data":"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"}
Sep 10 11:43:25.635349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.635276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7" event={"ID":"647cc619-9dfd-4913-9f78-368bc5de76e9","Type":"ContainerDied","Data":"8be027c25bd7cc06c9e88eb4126c6691cb644095ab1fa91ec8ddfc311629fd7f"}
Sep 10 11:43:25.635349 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.635299    2578 scope.go:117] "RemoveContainer" containerID="f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"
Sep 10 11:43:25.645612 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.645588    2578 scope.go:117] "RemoveContainer" containerID="f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"
Sep 10 11:43:25.646038 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:43:25.646015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2\": container with ID starting with f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2 not found: ID does not exist" containerID="f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"
Sep 10 11:43:25.646151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.646047    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2"} err="failed to get container status \"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2\": rpc error: code = NotFound desc = could not find container \"f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2\": container with ID starting with f68b5200d61687f1c185f52db275bb1877cbbbea1211881021842db3815295c2 not found: ID does not exist"
Sep 10 11:43:25.670853 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.670827    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:43:25.677731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:25.677703    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13c392-d496bcc8-xgbb7"]
Sep 10 11:43:26.455835 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:26.455798    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647cc619-9dfd-4913-9f78-368bc5de76e9" path="/var/lib/kubelet/pods/647cc619-9dfd-4913-9f78-368bc5de76e9/volumes"
Sep 10 11:43:32.241150 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:32.241118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:32.341987 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:32.341946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:32.498151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:32.498066    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:43:32.498323 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:32.498297    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy" containerID="cri-o://b3a9c2b50125c1a8261dc3fe9785c45a686cc7fe85ccff7d23397324479c1716" gracePeriod=30
Sep 10 11:43:32.542542 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:32.542506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:43:37.698941 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.698907    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerID="b3a9c2b50125c1a8261dc3fe9785c45a686cc7fe85ccff7d23397324479c1716" exitCode=0
Sep 10 11:43:37.699385 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.698980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerDied","Data":"b3a9c2b50125c1a8261dc3fe9785c45a686cc7fe85ccff7d23397324479c1716"}
Sep 10 11:43:37.699385 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.699051    2578 scope.go:117] "RemoveContainer" containerID="75fedc94ae7674198a41a35c2008342f72817dd1ce81492fa45934ff1d55390b"
Sep 10 11:43:37.739155 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.739133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:43:37.829935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.829826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.829935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.829889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hmsq\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.829935 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.829923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.829952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.829984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830274 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs\") pod \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\" (UID: \"d5992ce9-2d91-4d35-96a1-3e461b8f2f20\") "
Sep 10 11:43:37.830577 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:43:37.830577 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:37.830577 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:37.830577 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:37.830753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:37.830753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830597    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istiod-ca-cert\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.830753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830615    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-data\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.830753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830625    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.830753 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.830633    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-workload-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.832364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.832339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 11:43:37.832468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.832446    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:37.832879 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.832857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq" (OuterVolumeSpecName: "kube-api-access-5hmsq") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "kube-api-access-5hmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:43:37.832879 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.832861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5992ce9-2d91-4d35-96a1-3e461b8f2f20" (UID: "d5992ce9-2d91-4d35-96a1-3e461b8f2f20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:43:37.931442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.931407    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-token\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.931442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.931435    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-credential-socket\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.931442 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.931445    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmsq\" (UniqueName: \"kubernetes.io/projected/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-kube-api-access-5hmsq\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.931688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.931454    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-podinfo\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:37.931688 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:37.931463    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5992ce9-2d91-4d35-96a1-3e461b8f2f20-istio-envoy\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:43:38.705520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:38.705492    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"
Sep 10 11:43:38.705942 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:38.705491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b" event={"ID":"d5992ce9-2d91-4d35-96a1-3e461b8f2f20","Type":"ContainerDied","Data":"210cfd55a5b82d8a8d534a7846a856fbacc9b16e59185d26f22d8aa8516f97d1"}
Sep 10 11:43:38.705942 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:38.705624    2578 scope.go:117] "RemoveContainer" containerID="b3a9c2b50125c1a8261dc3fe9785c45a686cc7fe85ccff7d23397324479c1716"
Sep 10 11:43:38.727955 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:38.727923    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:43:38.733048 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:38.733024    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13c392-openshift-default-7dfc846bd-5mb7b"]
Sep 10 11:43:40.455684 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:43:40.455653    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" path="/var/lib/kubelet/pods/d5992ce9-2d91-4d35-96a1-3e461b8f2f20/volumes"
Sep 10 11:44:42.536763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:42.536732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:44:42.539339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:42.539314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:44:42.850343 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:42.850259    2578 scope.go:117] "RemoveContainer" containerID="a30e91f1c479de5cd291adcbd8cafcd219b2174faa4e9d6efacb0eea6f0adf34"
Sep 10 11:44:42.858639 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:42.858611    2578 scope.go:117] "RemoveContainer" containerID="187633b875876256952c8310aa427478b94471fd2b9bdb5070320a54c0c5475c"
Sep 10 11:44:55.600099 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:55.600044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:44:56.548390 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:56.548357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:44:58.180128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:44:58.180076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:45:00.134165 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134132    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"]
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134456    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134468    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134480    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134485    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134504    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134514    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134518    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134524    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393b3251-99be-4684-b1b0-99643174fae0" containerName="payload-processing"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134529    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="393b3251-99be-4684-b1b0-99643174fae0" containerName="payload-processing"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134535    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" containerName="payload-pre-processing"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134540    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" containerName="payload-pre-processing"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.134554 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134556    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134565    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134570    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647cc619-9dfd-4913-9f78-368bc5de76e9" containerName="payload-pre-processing"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134584    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="647cc619-9dfd-4913-9f78-368bc5de76e9" containerName="payload-pre-processing"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134591    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134596    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134649    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134655    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134661    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5992ce9-2d91-4d35-96a1-3e461b8f2f20" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134666    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134672    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134677    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8622ddb5-3434-45a9-9d3d-3f3b6177d913" containerName="payload-pre-processing"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134688    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="647cc619-9dfd-4913-9f78-368bc5de76e9" containerName="payload-pre-processing"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134694    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="393b3251-99be-4684-b1b0-99643174fae0" containerName="payload-processing"
Sep 10 11:45:00.135158 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.134700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf898b53-2a29-4c6b-a333-7f4939db47cf" containerName="istio-proxy"
Sep 10 11:45:00.137673 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.137654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"
Sep 10 11:45:00.140919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.140896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-99z58\""
Sep 10 11:45:00.147263 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.147243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"
Sep 10 11:45:00.150910 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.150472    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"]
Sep 10 11:45:00.424744 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:00.424720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"]
Sep 10 11:45:00.427171 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:45:00.427144    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode501345c_1080_4295_83d0_c249f98a3442.slice/crio-a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8 WatchSource:0}: Error finding container a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8: Status 404 returned error can't find the container with id a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8
Sep 10 11:45:01.103943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:01.103904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755" event={"ID":"e501345c-1080-4295-83d0-c249f98a3442","Type":"ContainerStarted","Data":"a880432226af715d46ebcbae0aa6fceffd375b4c697d3d709673df936a356bd2"}
Sep 10 11:45:01.104147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:01.103966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755" event={"ID":"e501345c-1080-4295-83d0-c249f98a3442","Type":"ContainerStarted","Data":"a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8"}
Sep 10 11:45:01.123298 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:01.123245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755" podStartSLOduration=1.123227974 podStartE2EDuration="1.123227974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:45:01.122309322 +0000 UTC m=+1819.197634980" watchObservedRunningTime="2026-09-10 11:45:01.123227974 +0000 UTC m=+1819.198553637"
Sep 10 11:45:02.111400 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:02.111368    2578 generic.go:356] "Generic (PLEG): container finished" podID="e501345c-1080-4295-83d0-c249f98a3442" containerID="a880432226af715d46ebcbae0aa6fceffd375b4c697d3d709673df936a356bd2" exitCode=0
Sep 10 11:45:02.111810 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:02.111446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755" event={"ID":"e501345c-1080-4295-83d0-c249f98a3442","Type":"ContainerDied","Data":"a880432226af715d46ebcbae0aa6fceffd375b4c697d3d709673df936a356bd2"}
Sep 10 11:45:03.239029 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:03.239004    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"
Sep 10 11:45:04.122661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:04.122622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755"
Sep 10 11:45:04.122661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:04.122650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817345-dk755" event={"ID":"e501345c-1080-4295-83d0-c249f98a3442","Type":"ContainerDied","Data":"a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8"}
Sep 10 11:45:04.122889 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:04.122695    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1a9884fbb3abe3f23303ea407e0b2fd02d1a20eb7faafe3c5f966458be2bcb8"
Sep 10 11:45:42.887024 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:45:42.886991    2578 scope.go:117] "RemoveContainer" containerID="8b4a35d43ff4ecf5ce899803b3f5b81ab2a4276bf3cd067924f2bfd1f0d37c77"
Sep 10 11:47:29.129108 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:47:29.129050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:47:51.985357 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:47:51.985323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:48:07.193482 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193748    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193759    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e78459-1e55-4479-b9e5-0dd69185a64c" containerName="istio-proxy"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e501345c-1080-4295-83d0-c249f98a3442" containerName="cleanup"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193773    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e501345c-1080-4295-83d0-c249f98a3442" containerName="cleanup"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193837    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24c7ac0-1b28-45bd-bca4-bf24b44d03a8" containerName="istio-proxy"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193849    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e501345c-1080-4295-83d0-c249f98a3442" containerName="cleanup"
Sep 10 11:48:07.193966 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.193856    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c215354-a78c-4cee-988f-b632ce0b27d3" containerName="istio-proxy"
Sep 10 11:48:07.196834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.196817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.207674 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.207651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f51e41c8-kserve-self-signed-certs\""
Sep 10 11:48:07.215126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.215102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:07.320720 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.320656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:48:07.388948 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.388913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.389147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.388954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xv7j\" (UniqueName: \"kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.389147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.389025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.389147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.389047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.389147 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.389076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.389321 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.389241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.490340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.490340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xv7j\" (UniqueName: \"kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.490340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.490340 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.491044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.491044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.491044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.491044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.491044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.490942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.492752 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.492728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.493013 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.492995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.500211 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.500183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xv7j\" (UniqueName: \"kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j\") pod \"e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") " pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.505989 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.505968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:07.781763 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.781731    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:07.783248 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:48:07.783223    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26702732_f847_44bc_8234_407d3e59b21b.slice/crio-c4295207bd5c464d3c0c876b4546c0d7460ab9772f49f5ccf056429c3235f140 WatchSource:0}: Error finding container c4295207bd5c464d3c0c876b4546c0d7460ab9772f49f5ccf056429c3235f140: Status 404 returned error can't find the container with id c4295207bd5c464d3c0c876b4546c0d7460ab9772f49f5ccf056429c3235f140
Sep 10 11:48:07.987042 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.986999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" event={"ID":"26702732-f847-44bc-8234-407d3e59b21b","Type":"ContainerStarted","Data":"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"}
Sep 10 11:48:07.987245 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:07.987051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" event={"ID":"26702732-f847-44bc-8234-407d3e59b21b","Type":"ContainerStarted","Data":"c4295207bd5c464d3c0c876b4546c0d7460ab9772f49f5ccf056429c3235f140"}
Sep 10 11:48:08.009827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:08.009679    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" podStartSLOduration=1.009660604 podStartE2EDuration="1.009660604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:48:08.009037924 +0000 UTC m=+2006.084363591" watchObservedRunningTime="2026-09-10 11:48:08.009660604 +0000 UTC m=+2006.084986268"
Sep 10 11:48:17.506525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:17.506468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:17.506525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:17.506530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:17.519403 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:17.519376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:18.045827 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:18.045798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:22.082010 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:22.081975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:48:28.966142 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:28.966075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:48:30.213265 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:30.213229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:30.237506 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:30.237473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:48:45.233078 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.233014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" podUID="26702732-f847-44bc-8234-407d3e59b21b" containerName="main" containerID="cri-o://5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab" gracePeriod=45
Sep 10 11:48:45.479411 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.479386    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:45.638220 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638220 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xv7j\" (UniqueName: \"kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638468 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs\") pod \"26702732-f847-44bc-8234-407d3e59b21b\" (UID: \"26702732-f847-44bc-8234-407d3e59b21b\") "
Sep 10 11:48:45.638676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache" (OuterVolumeSpecName: "model-cache") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:48:45.638676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:48:45.638676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home" (OuterVolumeSpecName: "home") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:48:45.638676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638632    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-model-cache\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:45.638676 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.638647    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-tmp-dir\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:45.640576 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.640553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm" (OuterVolumeSpecName: "dshm") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 11:48:45.640675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.640607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j" (OuterVolumeSpecName: "kube-api-access-8xv7j") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "kube-api-access-8xv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:48:45.640675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.640646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26702732-f847-44bc-8234-407d3e59b21b" (UID: "26702732-f847-44bc-8234-407d3e59b21b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:48:45.739856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.739815    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-home\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:45.739856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.739848    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26702732-f847-44bc-8234-407d3e59b21b-dshm\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:45.739856 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.739861    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xv7j\" (UniqueName: \"kubernetes.io/projected/26702732-f847-44bc-8234-407d3e59b21b-kube-api-access-8xv7j\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:45.740133 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:45.739876    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26702732-f847-44bc-8234-407d3e59b21b-tls-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:46.168731 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.168698    2578 generic.go:356] "Generic (PLEG): container finished" podID="26702732-f847-44bc-8234-407d3e59b21b" containerID="5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab" exitCode=0
Sep 10 11:48:46.168900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.168743    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"
Sep 10 11:48:46.168900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.168749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" event={"ID":"26702732-f847-44bc-8234-407d3e59b21b","Type":"ContainerDied","Data":"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"}
Sep 10 11:48:46.168900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.168781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6" event={"ID":"26702732-f847-44bc-8234-407d3e59b21b","Type":"ContainerDied","Data":"c4295207bd5c464d3c0c876b4546c0d7460ab9772f49f5ccf056429c3235f140"}
Sep 10 11:48:46.168900 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.168813    2578 scope.go:117] "RemoveContainer" containerID="5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"
Sep 10 11:48:46.177912 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.177893    2578 scope.go:117] "RemoveContainer" containerID="5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"
Sep 10 11:48:46.178188 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:48:46.178169    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab\": container with ID starting with 5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab not found: ID does not exist" containerID="5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"
Sep 10 11:48:46.178261 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.178200    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab"} err="failed to get container status \"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab\": rpc error: code = NotFound desc = could not find container \"5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab\": container with ID starting with 5f07cbeccfc1ff35de6c01aa3ef32d0845d311744df8c986969aec6a10a642ab not found: ID does not exist"
Sep 10 11:48:46.196249 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.196225    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:46.200962 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.200942    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f51e41c8-kserve-7cdf4f74dc-fmwx6"]
Sep 10 11:48:46.454802 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:46.454730    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26702732-f847-44bc-8234-407d3e59b21b" path="/var/lib/kubelet/pods/26702732-f847-44bc-8234-407d3e59b21b/volumes"
Sep 10 11:48:50.941211 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.941167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:48:50.941785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.941513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26702732-f847-44bc-8234-407d3e59b21b" containerName="main"
Sep 10 11:48:50.941785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.941526    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26702732-f847-44bc-8234-407d3e59b21b" containerName="main"
Sep 10 11:48:50.941785 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.941589    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26702732-f847-44bc-8234-407d3e59b21b" containerName="main"
Sep 10 11:48:50.946232 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.946208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:50.952001 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.951918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 11:48:50.952001 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.951930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gkwvn\""
Sep 10 11:48:50.952305 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.952037    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 11:48:50.952434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.952364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 11:48:50.952434 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.952418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 11:48:50.958450 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:50.958429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:48:51.087712 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.087674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.087919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.087716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk485\" (UniqueName: \"kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.087919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.087747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.087919 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.087853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.189308 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.189276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk485\" (UniqueName: \"kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.189472 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.189331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.189472 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.189405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.189588 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.189478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.190053 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.190023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.192379 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.192307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.192514 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.192378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.197702 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.197677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk485\" (UniqueName: \"kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485\") pod \"maas-api-5c55758554-5bjsp\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") " pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.257471 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.257429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:51.530832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.530806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:48:51.532683 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:48:51.532657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c1a13_63eb_42f2_a126_942781610850.slice/crio-5da05eed2acb329bccb1b72184dc1a935fc8be9bc68a88c17074318fbbabb70a WatchSource:0}: Error finding container 5da05eed2acb329bccb1b72184dc1a935fc8be9bc68a88c17074318fbbabb70a: Status 404 returned error can't find the container with id 5da05eed2acb329bccb1b72184dc1a935fc8be9bc68a88c17074318fbbabb70a
Sep 10 11:48:51.534015 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:51.533991    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:48:52.201611 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:52.201572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" event={"ID":"ea2c1a13-63eb-42f2-a126-942781610850","Type":"ContainerStarted","Data":"5da05eed2acb329bccb1b72184dc1a935fc8be9bc68a88c17074318fbbabb70a"}
Sep 10 11:48:54.222377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:54.222279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" event={"ID":"ea2c1a13-63eb-42f2-a126-942781610850","Type":"ContainerStarted","Data":"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"}
Sep 10 11:48:54.222782 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:54.222407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:54.244659 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:54.244608    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" podStartSLOduration=1.808051825 podStartE2EDuration="4.244591845s" totalImagesPullingTime="2.43654002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:48:54.243202409 +0000 UTC m=+2052.318528074" watchObservedRunningTime="2026-09-10 11:48:54.244591845 +0000 UTC m=+2052.319917573"
Sep 10 11:48:58.779840 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:58.779798    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:48:58.780347 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:58.780027    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" podUID="ea2c1a13-63eb-42f2-a126-942781610850" containerName="maas-api" containerID="cri-o://9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3" gracePeriod=30
Sep 10 11:48:59.023956 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.023934    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:59.054376 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls\") pod \"ea2c1a13-63eb-42f2-a126-942781610850\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") "
Sep 10 11:48:59.054376 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca\") pod \"ea2c1a13-63eb-42f2-a126-942781610850\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") "
Sep 10 11:48:59.054592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs\") pod \"ea2c1a13-63eb-42f2-a126-942781610850\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") "
Sep 10 11:48:59.054592 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk485\" (UniqueName: \"kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485\") pod \"ea2c1a13-63eb-42f2-a126-942781610850\" (UID: \"ea2c1a13-63eb-42f2-a126-942781610850\") "
Sep 10 11:48:59.054745 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea2c1a13-63eb-42f2-a126-942781610850" (UID: "ea2c1a13-63eb-42f2-a126-942781610850"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:48:59.054851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.054835    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c1a13-63eb-42f2-a126-942781610850-openshift-service-ca\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:59.056832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.056805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea2c1a13-63eb-42f2-a126-942781610850" (UID: "ea2c1a13-63eb-42f2-a126-942781610850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:48:59.056832 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.056810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485" (OuterVolumeSpecName: "kube-api-access-xk485") pod "ea2c1a13-63eb-42f2-a126-942781610850" (UID: "ea2c1a13-63eb-42f2-a126-942781610850"). InnerVolumeSpecName "kube-api-access-xk485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:48:59.056969 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.056943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea2c1a13-63eb-42f2-a126-942781610850" (UID: "ea2c1a13-63eb-42f2-a126-942781610850"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:48:59.155629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.155597    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk485\" (UniqueName: \"kubernetes.io/projected/ea2c1a13-63eb-42f2-a126-942781610850-kube-api-access-xk485\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:59.155629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.155624    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-maas-api-tls\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:59.155629 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.155634    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c1a13-63eb-42f2-a126-942781610850-metrics-certs\") on node \"ip-10-0-141-40.ec2.internal\" DevicePath \"\""
Sep 10 11:48:59.246058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.246025    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea2c1a13-63eb-42f2-a126-942781610850" containerID="9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3" exitCode=0
Sep 10 11:48:59.246058 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.246063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" event={"ID":"ea2c1a13-63eb-42f2-a126-942781610850","Type":"ContainerDied","Data":"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"}
Sep 10 11:48:59.246364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.246071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"
Sep 10 11:48:59.246364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.246137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c55758554-5bjsp" event={"ID":"ea2c1a13-63eb-42f2-a126-942781610850","Type":"ContainerDied","Data":"5da05eed2acb329bccb1b72184dc1a935fc8be9bc68a88c17074318fbbabb70a"}
Sep 10 11:48:59.246364 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.246160    2578 scope.go:117] "RemoveContainer" containerID="9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"
Sep 10 11:48:59.255617 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.255588    2578 scope.go:117] "RemoveContainer" containerID="9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"
Sep 10 11:48:59.255913 ip-10-0-141-40 kubenswrapper[2578]: E0910 11:48:59.255893    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3\": container with ID starting with 9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3 not found: ID does not exist" containerID="9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"
Sep 10 11:48:59.256006 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.255923    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3"} err="failed to get container status \"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3\": rpc error: code = NotFound desc = could not find container \"9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3\": container with ID starting with 9832cf0b832955385816aa73061d240d03d73b3073b2a91f3bf3b282379bd4a3 not found: ID does not exist"
Sep 10 11:48:59.269805 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.269779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:48:59.274315 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:48:59.274292    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c55758554-5bjsp"]
Sep 10 11:49:00.455852 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:00.455821    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2c1a13-63eb-42f2-a126-942781610850" path="/var/lib/kubelet/pods/ea2c1a13-63eb-42f2-a126-942781610850/volumes"
Sep 10 11:49:20.949834 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:20.949754    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-646d785fb4-fb5pd_4eaeef07-98e6-4bad-901e-f111fc1d9aad/authorino/0.log"
Sep 10 11:49:26.764862 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:26.764827    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m9c44_1c100be0-eb1e-402a-914b-be6286228675/manager/0.log"
Sep 10 11:49:27.442211 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:27.442177    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d6d6998d4-nqd6s_5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5/copy-manifests/0.log"
Sep 10 11:49:27.455525 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:27.455495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d6d6998d4-nqd6s_5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5/manager/0.log"
Sep 10 11:49:28.299903 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:28.299876    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817345-dk755_e501345c-1080-4295-83d0-c249f98a3442/cleanup/0.log"
Sep 10 11:49:29.261005 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:29.260954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-646d785fb4-fb5pd_4eaeef07-98e6-4bad-901e-f111fc1d9aad/authorino/0.log"
Sep 10 11:49:29.397291 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:29.397258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6lv9g_d0ebd4e8-ba22-4c92-bf2b-2987c2516708/manager/0.log"
Sep 10 11:49:29.650661 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:29.650640    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-4kzpg_477c6af0-0e3d-4fde-90ab-f884dfa16d92/kuadrant-console-plugin/0.log"
Sep 10 11:49:30.415318 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:30.415287    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6_fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4/istio-proxy/0.log"
Sep 10 11:49:30.947342 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:30.947311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/1.log"
Sep 10 11:49:31.703943 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:31.703912    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nzl_89e77b37-d395-4e21-ad01-c796c8cd4282/main/0.log"
Sep 10 11:49:31.948044 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:31.948011    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8t79z_3af934cc-954d-473d-85f5-755b8c54b553/main/0.log"
Sep 10 11:49:38.031563 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.031525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-79gdg/must-gather-xj9mr"]
Sep 10 11:49:38.031950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.031836    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2c1a13-63eb-42f2-a126-942781610850" containerName="maas-api"
Sep 10 11:49:38.031950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.031847    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2c1a13-63eb-42f2-a126-942781610850" containerName="maas-api"
Sep 10 11:49:38.031950 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.031904    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2c1a13-63eb-42f2-a126-942781610850" containerName="maas-api"
Sep 10 11:49:38.038534 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.038510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.041716 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.041695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-79gdg\"/\"default-dockercfg-rl99x\""
Sep 10 11:49:38.042195 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.042178    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-79gdg\"/\"openshift-service-ca.crt\""
Sep 10 11:49:38.042446 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.042427    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-79gdg\"/\"kube-root-ca.crt\""
Sep 10 11:49:38.056914 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.056880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79gdg/must-gather-xj9mr"]
Sep 10 11:49:38.082126 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.082076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:49:38.172538 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.172505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log" limit=1024
Sep 10 11:49:38.191682 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.191645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4fa30172-4c59-4425-b4e3-42607c61ce47-must-gather-output\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.191840 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.191705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdwx\" (UniqueName: \"kubernetes.io/projected/4fa30172-4c59-4425-b4e3-42607c61ce47-kube-api-access-7hdwx\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.293125 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.293023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4fa30172-4c59-4425-b4e3-42607c61ce47-must-gather-output\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.293125 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.293098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdwx\" (UniqueName: \"kubernetes.io/projected/4fa30172-4c59-4425-b4e3-42607c61ce47-kube-api-access-7hdwx\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.293422 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.293401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4fa30172-4c59-4425-b4e3-42607c61ce47-must-gather-output\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.302820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.302791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdwx\" (UniqueName: \"kubernetes.io/projected/4fa30172-4c59-4425-b4e3-42607c61ce47-kube-api-access-7hdwx\") pod \"must-gather-xj9mr\" (UID: \"4fa30172-4c59-4425-b4e3-42607c61ce47\") " pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.347528 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.347495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79gdg/must-gather-xj9mr"
Sep 10 11:49:38.628467 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:38.628439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79gdg/must-gather-xj9mr"]
Sep 10 11:49:38.630312 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:49:38.630286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fa30172_4c59_4425_b4e3_42607c61ce47.slice/crio-4dbeae37ae6419364c94546fc5f1c336b484382676227da0373cdf5e4e7a763a WatchSource:0}: Error finding container 4dbeae37ae6419364c94546fc5f1c336b484382676227da0373cdf5e4e7a763a: Status 404 returned error can't find the container with id 4dbeae37ae6419364c94546fc5f1c336b484382676227da0373cdf5e4e7a763a
Sep 10 11:49:39.436711 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:39.436676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79gdg/must-gather-xj9mr" event={"ID":"4fa30172-4c59-4425-b4e3-42607c61ce47","Type":"ContainerStarted","Data":"4dbeae37ae6419364c94546fc5f1c336b484382676227da0373cdf5e4e7a763a"}
Sep 10 11:49:40.445549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:40.444652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79gdg/must-gather-xj9mr" event={"ID":"4fa30172-4c59-4425-b4e3-42607c61ce47","Type":"ContainerStarted","Data":"5a46f50ddd4ad41ec04f0234794f9bc944f91c65a59364bd48d258ea7d7f6a37"}
Sep 10 11:49:40.445549 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:40.444712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79gdg/must-gather-xj9mr" event={"ID":"4fa30172-4c59-4425-b4e3-42607c61ce47","Type":"ContainerStarted","Data":"b9919e81cb1ec4385d0f6df4a8fece80f548e1e52208996510a7e257ab55f1ba"}
Sep 10 11:49:40.466551 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:40.466486    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-79gdg/must-gather-xj9mr" podStartSLOduration=1.595040059 podStartE2EDuration="2.466461512s" totalImagesPullingTime="871.421453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:49:40.464684313 +0000 UTC m=+2098.540010003" watchObservedRunningTime="2026-09-10 11:49:40.466461512 +0000 UTC m=+2098.541787176"
Sep 10 11:49:41.213912 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:41.213872    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dh54m_34887360-7db4-44aa-8c7a-976d77303b5b/global-pull-secret-syncer/0.log"
Sep 10 11:49:41.332418 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:41.332373    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-794tn_aaa45f9c-d6a7-4a89-b4df-4048ea79d0eb/konnectivity-agent/0.log"
Sep 10 11:49:41.491007 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:41.490938    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-40.ec2.internal_3d6d72dcc0110b2d026681ad98a3f85b/haproxy/0.log"
Sep 10 11:49:42.587626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:42.587573    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:49:42.588934 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:42.588903    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:49:46.146507 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:46.145853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-646d785fb4-fb5pd_4eaeef07-98e6-4bad-901e-f111fc1d9aad/authorino/0.log"
Sep 10 11:49:46.187060 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:46.187022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6lv9g_d0ebd4e8-ba22-4c92-bf2b-2987c2516708/manager/0.log"
Sep 10 11:49:46.254985 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:46.254947    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-4kzpg_477c6af0-0e3d-4fde-90ab-f884dfa16d92/kuadrant-console-plugin/0.log"
Sep 10 11:49:48.140037 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:48.139998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-6drlq_9e478389-5b0b-44e8-a48a-3eb07b761b8b/cluster-monitoring-operator/0.log"
Sep 10 11:49:48.547226 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:48.547194    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtwh4_6b671393-bc0d-4047-bdf7-2da219e22717/node-exporter/0.log"
Sep 10 11:49:48.580888 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:48.580861    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtwh4_6b671393-bc0d-4047-bdf7-2da219e22717/kube-rbac-proxy/0.log"
Sep 10 11:49:48.611675 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:48.611583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtwh4_6b671393-bc0d-4047-bdf7-2da219e22717/init-textfile/0.log"
Sep 10 11:49:50.699251 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:50.699214    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-rtzz2_e175bf87-32d0-422a-b5ea-4dd6a5b58fb5/networking-console-plugin/0.log"
Sep 10 11:49:51.398205 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.398172    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/2.log"
Sep 10 11:49:51.406134 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.406079    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-glrhj_f59e35f6-4a05-4a79-86fd-f5aeaf409d61/console-operator/3.log"
Sep 10 11:49:51.418248 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.418217    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"]
Sep 10 11:49:51.426116 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.426074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.443587 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.443552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"]
Sep 10 11:49:51.538151 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.538109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grfh\" (UniqueName: \"kubernetes.io/projected/8aeca126-c937-40b0-9b07-846969bdf460-kube-api-access-8grfh\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.538350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.538199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-lib-modules\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.538350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.538265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-podres\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.538350 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.538289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-sys\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.538520 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.538413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-proc\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639293 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-proc\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grfh\" (UniqueName: \"kubernetes.io/projected/8aeca126-c937-40b0-9b07-846969bdf460-kube-api-access-8grfh\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-lib-modules\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-proc\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-podres\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639474 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-sys\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-lib-modules\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-podres\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.639734 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.639560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aeca126-c937-40b0-9b07-846969bdf460-sys\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.650480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.650406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grfh\" (UniqueName: \"kubernetes.io/projected/8aeca126-c937-40b0-9b07-846969bdf460-kube-api-access-8grfh\") pod \"perf-node-gather-daemonset-7s4j7\" (UID: \"8aeca126-c937-40b0-9b07-846969bdf460\") " pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:51.742552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:51.742506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:52.042820 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:52.042795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"]
Sep 10 11:49:52.046306 ip-10-0-141-40 kubenswrapper[2578]: W0910 11:49:52.046278    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8aeca126_c937_40b0_9b07_846969bdf460.slice/crio-47c6fd4246487ef3fe8d74540605d5236f02f70b84389a9fd42bc310bcd5c29b WatchSource:0}: Error finding container 47c6fd4246487ef3fe8d74540605d5236f02f70b84389a9fd42bc310bcd5c29b: Status 404 returned error can't find the container with id 47c6fd4246487ef3fe8d74540605d5236f02f70b84389a9fd42bc310bcd5c29b
Sep 10 11:49:52.545854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:52.545150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7" event={"ID":"8aeca126-c937-40b0-9b07-846969bdf460","Type":"ContainerStarted","Data":"30d5381d3ec7b2211ae5991261621d130a02e708eef65b01c6b989d21bc9829c"}
Sep 10 11:49:52.545854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:52.545227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7" event={"ID":"8aeca126-c937-40b0-9b07-846969bdf460","Type":"ContainerStarted","Data":"47c6fd4246487ef3fe8d74540605d5236f02f70b84389a9fd42bc310bcd5c29b"}
Sep 10 11:49:52.545854 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:52.545497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:49:52.567128 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:52.567041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7" podStartSLOduration=1.567018929 podStartE2EDuration="1.567018929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:49:52.564989814 +0000 UTC m=+2110.640315479" watchObservedRunningTime="2026-09-10 11:49:52.567018929 +0000 UTC m=+2110.642344594"
Sep 10 11:49:54.083290 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:54.083258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbfxf_5e3b4414-485e-4bf7-8743-c0b0b6a47226/dns/0.log"
Sep 10 11:49:54.116626 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:54.116592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbfxf_5e3b4414-485e-4bf7-8743-c0b0b6a47226/kube-rbac-proxy/0.log"
Sep 10 11:49:54.205383 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:54.205354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bbh66_8b13de39-1c19-4465-9e6c-2d077b260bd5/dns-node-resolver/0.log"
Sep 10 11:49:54.858846 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:54.858817    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm6c6_85c3fa39-d36c-470b-99f0-14820da3b3dd/node-ca/0.log"
Sep 10 11:49:55.939552 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:55.939525    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6_fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4/istio-proxy/0.log"
Sep 10 11:49:56.456378 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:56.456347    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/0.log"
Sep 10 11:49:56.467719 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:56.467657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/1.log"
Sep 10 11:49:57.583930 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:57.583894    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tq59_c493c32a-781d-4ced-9053-f22d34da37af/serve-healthcheck-canary/0.log"
Sep 10 11:49:58.157769 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.157729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-8j76t_2dc89937-563f-4148-a495-37312d5beea1/insights-operator/0.log"
Sep 10 11:49:58.158372 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.158331    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-8j76t_2dc89937-563f-4148-a495-37312d5beea1/insights-operator/1.log"
Sep 10 11:49:58.360027 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.359998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5wlr_c5ec7ff7-d754-48ce-a296-be92da83561f/kube-rbac-proxy/0.log"
Sep 10 11:49:58.387181 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.387153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5wlr_c5ec7ff7-d754-48ce-a296-be92da83561f/exporter/0.log"
Sep 10 11:49:58.419801 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.419702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5wlr_c5ec7ff7-d754-48ce-a296-be92da83561f/extractor/0.log"
Sep 10 11:49:58.564480 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:49:58.564451    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-79gdg/perf-node-gather-daemonset-7s4j7"
Sep 10 11:50:01.134140 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:01.134072    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m9c44_1c100be0-eb1e-402a-914b-be6286228675/manager/0.log"
Sep 10 11:50:01.515377 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:01.515303    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d6d6998d4-nqd6s_5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5/manager/0.log"
Sep 10 11:50:01.588851 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:01.588820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d6d6998d4-nqd6s_5cd4ba6f-d8a1-4c8f-9f5c-5f414d3ce1e5/copy-manifests/0.log"
Sep 10 11:50:02.006339 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:02.006310    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6_fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4/istio-proxy/0.log"
Sep 10 11:50:02.487627 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:02.487592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/0.log"
Sep 10 11:50:02.498727 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:02.498692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/1.log"
Sep 10 11:50:03.071049 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:03.071016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b89dcdb5f-4bnj2_e3784c54-65cb-49f8-af70-37fbbcffce9b/manager/0.log"
Sep 10 11:50:03.141636 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:03.141607    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hvrkv_79ba8d4f-36db-4b78-862a-6db07c559f8e/openshift-lws-operator/0.log"
Sep 10 11:50:04.166970 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:04.166939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x5kr6_fc7d4419-a9e1-47f6-ac98-5a87fb6b3ee4/istio-proxy/0.log"
Sep 10 11:50:04.709636 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:04.709607    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/0.log"
Sep 10 11:50:04.719063 ip-10-0-141-40 kubenswrapper[2578]: I0910 11:50:04.719036    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4cj64_2df9ea96-1641-4fba-8a1d-3000572c2b80/istio-proxy/1.log"